From ada14788f62c54ba73df2a355a56699407fa70df Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Wed, 12 Aug 2026 17:49:15 -0400 Subject: [PATCH 01/27] feat(helm): add optional BackendTLSPolicy for e2e TLS Add grpcRoute.backendTLSPolicy values to optionally create a BackendTLSPolicy resource that enables end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The Gateway proxy terminates client-facing TLS and re-encrypts when connecting to the backend, validating the pod's certificate against a user-supplied CA ConfigMap. This removes the requirement to set server.disableTls=true when using HTTPS at the Gateway listener. Supported on OpenShift 4.22+ and other platforms with BackendTLSPolicy support in the Gateway API implementation. Update OpenShift and ingress documentation with e2e TLS instructions. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/README.md | 6 + deploy/helm/openshell/README.md.gotmpl | 3 + .../templates/backend-tls-policy.yaml | 29 ++++ deploy/helm/openshell/templates/gateway.yaml | 4 +- deploy/helm/openshell/values.yaml | 23 +++ docs/kubernetes/ingress.mdx | 47 +++++++ docs/kubernetes/openshift.mdx | 131 +++++++++++++++++- 7 files changed, 235 insertions(+), 8 deletions(-) create mode 100644 deploy/helm/openshell/templates/backend-tls-policy.yaml diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 92bd2a9db8..2b2efde9a2 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -56,6 +56,9 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version ..svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | | grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index cf8677741e..6e6f2012fc 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -56,6 +56,9 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version ..svc.cluster.local) when empty, which matches + # the SAN included by both cert-manager and the pkiInitJob. + hostname: "" # OpenShift Route with TLS passthrough. The gateway terminates its own # TLS/mTLS; the router only forwards based on SNI, so it never sees plaintext diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 51284465d9..cecd4b1687 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -135,6 +135,53 @@ openshell status See [Authentication](/kubernetes/setup) for OIDC issuer, audience, and roles configuration. +## End-to-end TLS (BackendTLSPolicy) + +As an alternative to the plaintext backend path above, the chart can create a `BackendTLSPolicy` that tells the Gateway proxy to re-encrypt traffic when connecting to the OpenShell gateway pod: + +```text +client → HTTPS → Gateway (terminate TLS) → TLS (re-encrypt) → openshell gateway pod +``` + +This keeps TLS on the gateway pod rather than disabling it with `server.disableTls=true`. The Gateway proxy validates the backend's certificate against a CA you supply in a ConfigMap. + +BackendTLSPolicy is a standard Gateway API resource. It is supported on OpenShift 4.22+ (via the OpenShift gateway controller) and on other platforms where the Gateway API implementation supports it (check your controller's documentation). + +### Create the backend CA ConfigMap + +After the server certificate is created (by cert-manager or the pkiInitJob), extract the CA certificate into a ConfigMap: + +```shell +kubectl -n openshell get secret openshell-server-tls \ + -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt + +kubectl -n openshell create configmap openshell-backend-ca \ + --from-file=ca.crt=/tmp/ca.crt +``` + +### Install with e2e TLS + +```shell +helm upgrade --install openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + --set grpcRoute.enabled=true \ + --set grpcRoute.gateway.create=true \ + --set grpcRoute.gateway.className=eg \ + --set grpcRoute.gateway.listener.protocol=HTTPS \ + --set grpcRoute.gateway.listener.port=443 \ + --set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \ + --set grpcRoute.backendTLSPolicy.enabled=true \ + --set grpcRoute.backendTLSPolicy.caCertificateConfigMapName=openshell-backend-ca \ + --set server.oidc.issuer=https:// \ + --set 'grpcRoute.hostnames[0]=' +``` + +Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. + +For OpenShift 4.22+, see [OpenShift](/kubernetes/openshift#end-to-end-tls-openshift-422) for platform-specific instructions including Gateway and GatewayClass setup. + ## SSH Relay Sandbox SSH uses the gateway endpoint registered with the CLI. No separate Helm SSH host or port values are required. diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 43e7d0338b..b14f5a0751 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -87,13 +87,132 @@ openshell gateway add http://127.0.0.1:8080 --local --name openshift openshell status ``` -## Production: expose externally with a real certificate +## Production Deployments -The steps above run the gateway over plaintext HTTP for quick evaluation. For -a real deployment, cert-manager can issue the gateway's server certificate -from a real Issuer or ClusterIssuer (for example, an ACME issuer), and an -OpenShift Route with TLS passthrough exposes it externally while the gateway -keeps terminating its own TLS and mTLS. +The steps above run the gateway over plaintext HTTP for quick evaluation. For production deployments, choose one of the approaches below based on your OpenShift version. + +### End-to-end TLS (OpenShift 4.22+) + +OpenShift 4.22 and later support `BackendTLSPolicy` in the Gateway API, enabling end-to-end TLS between the OpenShift router and the OpenShell gateway pod. The traffic flow is: + +```text +client → HTTPS → OpenShift Gateway (terminate TLS) → TLS (re-encrypt) → openshell gateway pod +``` + +This removes the requirement to run the gateway with `server.disableTls=true`. The OpenShift router terminates client-facing TLS at the listener and re-encrypts when connecting to the backend service, validating the backend's certificate against a CA you provide. + +#### Prerequisites + +- OpenShift 4.22+ cluster with the Gateway API enabled +- cert-manager installed (recommended) or the built-in pkiInitJob for server certificates +- A `GatewayClass` registered for the OpenShift gateway controller + +#### Create the GatewayClass + +If your cluster does not already have an OpenShift GatewayClass, create one: + +```shell +oc apply -f - <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: GatewayClass +metadata: + name: openshift-default +spec: + controllerName: openshift.io/gateway-controller/v1 +EOF +``` + +#### Create the Gateway + +Create a Gateway resource in the `openshift-ingress` namespace. Replace `` with your cluster's route hostname: + +```shell +oc apply -f - <<'EOF' +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: openshell-gateway + namespace: openshift-ingress +spec: + gatewayClassName: openshift-default + listeners: + - name: grpc + hostname: "" + port: 443 + protocol: HTTPS + tls: + mode: Terminate + certificateRefs: + - name: + kind: Secret + allowedRoutes: + namespaces: + from: Selector + selector: + matchLabels: + kubernetes.io/metadata.name: openshell +EOF +``` + +The listener TLS Secret should contain the certificate for the external hostname. On ROSA and managed OpenShift clusters, you can copy the cluster's wildcard certificate from the `openshift-ingress` namespace. + +#### Install with e2e TLS + +Install the chart with cert-manager, the GRPCRoute, and BackendTLSPolicy enabled. The server certificate SAN list must include the external hostname so the gateway can serve TLS to both the cluster-internal service FQDN and external clients: + +```shell +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + --set podSecurityContext.fsGroup=null \ + --set securityContext.runAsUser=null \ + --set certManager.enabled=true \ + --set 'certManager.serverDnsNames[0]=openshell' \ + --set 'certManager.serverDnsNames[1]=openshell.openshell.svc' \ + --set 'certManager.serverDnsNames[2]=openshell.openshell.svc.cluster.local' \ + --set 'certManager.serverDnsNames[3]=localhost' \ + --set 'certManager.serverDnsNames[4]=' \ + --set grpcRoute.enabled=true \ + --set grpcRoute.gateway.name=openshell-gateway \ + --set grpcRoute.gateway.namespace=openshift-ingress \ + --set 'grpcRoute.hostnames[0]=' \ + --set grpcRoute.backendTLSPolicy.enabled=true \ + --set grpcRoute.backendTLSPolicy.caCertificateConfigMapName=openshell-backend-ca +``` + +| Override | Reason | +|---|---| +| `podSecurityContext.fsGroup=null` / `securityContext.runAsUser=null` | Let OpenShift's SCC admission assign UIDs. | +| `certManager.enabled=true` | Use cert-manager for automatic server certificate lifecycle. | +| `grpcRoute.enabled=true` | Create a GRPCRoute pointing at the external Gateway. | +| `grpcRoute.gateway.name` / `namespace` | Reference the Gateway created above in `openshift-ingress`. | +| `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. | +| `grpcRoute.backendTLSPolicy.caCertificateConfigMapName` | ConfigMap containing the CA that signed the server certificate. | + +Note that `server.disableTls` is **not** set — the gateway pod serves TLS, and the BackendTLSPolicy tells the OpenShift router to connect over TLS. + +#### Create the backend CA ConfigMap + +After cert-manager issues the server certificate, create a ConfigMap containing the CA certificate so the BackendTLSPolicy can validate the backend: + +```shell +oc -n openshell get secret openshell-server-tls \ + -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt + +oc -n openshell create configmap openshell-backend-ca \ + --from-file=ca.crt=/tmp/ca.crt +``` + +#### Register over HTTPS + +```shell +openshell gateway add https:// --name openshift +openshell status +``` + +### Route with TLS Passthrough (All OpenShift Versions) + +For OpenShift versions prior to 4.22, or when you prefer Route-based ingress, cert-manager can issue the gateway's server certificate from a real Issuer or ClusterIssuer (for example, an ACME issuer), and an OpenShift Route with TLS passthrough exposes it externally while the gateway keeps terminating its own TLS and mTLS. Install cert-manager and configure a working `ClusterIssuer` first — see [Managing Certificates](/kubernetes/managing-certificates) for the From 3dc87add07238f676d38b3a547a94bacf190d279 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Thu, 13 Aug 2026 11:00:28 -0400 Subject: [PATCH 02/27] feat(helm,server): auto-create backend CA ConfigMap in certgen hook Extend the generate-certs command with --backend-ca-configmap-name and --backend-ca-source-secret flags. When BackendTLSPolicy is enabled, the certgen pre-install hook creates the CA ConfigMap automatically: - pkiInitJob mode (default): uses the CA from the generated PKI bundle. Fully automatic on first install. - cert-manager mode: reads ca.crt from the server TLS Secret. On first install the Secret does not exist yet (cert-manager reconciles after templates are applied), so the ConfigMap is created on the first helm upgrade. Logs a warning on the initial skip. The caCertificateConfigMapName value now defaults to -backend-ca when empty, so users only need to set backendTLSPolicy.enabled=true. Update certgen RBAC to include configmaps get/create when the feature is enabled. Add CLI arg parsing tests for the new flags. Signed-off-by: Brandon Squizzato --- crates/openshell-server/src/certgen.rs | 115 +++++++++++++++++- crates/openshell-server/src/cli.rs | 53 ++++++++ deploy/helm/openshell/README.md | 2 +- deploy/helm/openshell/templates/_helpers.tpl | 7 ++ .../templates/backend-tls-policy.yaml | 5 +- deploy/helm/openshell/templates/certgen.yaml | 11 ++ deploy/helm/openshell/values.yaml | 9 +- docs/kubernetes/ingress.mdx | 15 +-- docs/kubernetes/openshift.mdx | 27 +--- 9 files changed, 199 insertions(+), 45 deletions(-) diff --git a/crates/openshell-server/src/certgen.rs b/crates/openshell-server/src/certgen.rs index 1f3ffca02d..01c920242a 100644 --- a/crates/openshell-server/src/certgen.rs +++ b/crates/openshell-server/src/certgen.rs @@ -24,7 +24,7 @@ use clap::Args; use k8s_openapi::ByteString; -use k8s_openapi::api::core::v1::Secret; +use k8s_openapi::api::core::v1::{ConfigMap, Secret}; use kube::Client; use kube::api::{Api, ObjectMeta, PostParams}; use miette::{IntoDiagnostic, Result, WrapErr}; @@ -78,6 +78,20 @@ pub struct CertgenArgs { /// For local debugging. #[arg(long)] dry_run: bool, + + /// Name of a ConfigMap to create containing the CA certificate (key: ca.crt) + /// for BackendTLSPolicy backend validation. In full PKI mode, the CA comes + /// from the generated bundle. In --jwt-only mode, the CA is read from + /// --backend-ca-source-secret. + #[arg(long, value_name = "NAME")] + backend_ca_configmap_name: Option, + + /// Name of an existing Secret containing a ca.crt key to populate the + /// backend CA ConfigMap from. Required with --jwt-only when + /// --backend-ca-configmap-name is set (typically the server TLS Secret + /// created by cert-manager). + #[arg(long, value_name = "NAME", requires = "backend_ca_configmap_name")] + backend_ca_source_secret: Option, } pub async fn run(args: CertgenArgs) -> Result<()> { @@ -97,7 +111,13 @@ pub async fn run(args: CertgenArgs) -> Result<()> { run_local(dir, &args.server_sans) } else { let bundle = generate_pki(&args.server_sans)?; - run_kubernetes(&args, &bundle).await + run_kubernetes(&args, &bundle).await?; + + if let Some(ref cm_name) = args.backend_ca_configmap_name { + create_backend_ca_configmap_if_needed(&args, &bundle, cm_name).await?; + } + + Ok(()) } } @@ -293,6 +313,97 @@ async fn create_tls_secrets( Ok(()) } +async fn create_backend_ca_configmap_if_needed( + args: &CertgenArgs, + bundle: &PkiBundle, + configmap_name: &str, +) -> Result<()> { + let namespace = args + .namespace + .as_deref() + .ok_or_else(|| miette::miette!("--namespace is required (or set POD_NAMESPACE)"))?; + + let client = Client::try_default() + .await + .into_diagnostic() + .wrap_err("failed to construct Kubernetes client for backend CA ConfigMap")?; + let api: Api = Api::namespaced(client.clone(), namespace); + + if api + .get_opt(configmap_name) + .await + .into_diagnostic() + .wrap_err_with(|| format!("failed to read configmap {configmap_name}"))? + .is_some() + { + info!( + namespace = %namespace, + configmap = %configmap_name, + "Backend CA ConfigMap already exists, skipping." + ); + return Ok(()); + } + + let ca_pem = if !args.jwt_only { + bundle.ca_cert_pem.clone() + } else if let Some(source_secret) = &args.backend_ca_source_secret { + let secret_api: Api = Api::namespaced(client, namespace); + match secret_api + .get_opt(source_secret) + .await + .into_diagnostic() + .wrap_err_with(|| format!("failed to read secret {source_secret}"))? + { + Some(secret) => { + let data = secret.data.ok_or_else(|| { + miette::miette!("secret {source_secret} has no data") + })?; + let ca = data.get("ca.crt").ok_or_else(|| { + miette::miette!("secret {source_secret} has no ca.crt key") + })?; + String::from_utf8(ca.0.clone()) + .into_diagnostic() + .wrap_err("ca.crt is not valid UTF-8")? + } + None => { + warn!( + secret = %source_secret, + configmap = %configmap_name, + "Backend CA source secret not found; ConfigMap not created. \ + Create it manually or run helm upgrade after the TLS secret exists." + ); + return Ok(()); + } + } + } else { + return Err(miette::miette!( + "--backend-ca-source-secret is required with --jwt-only \ + and --backend-ca-configmap-name" + )); + }; + + let configmap = ConfigMap { + metadata: ObjectMeta { + name: Some(configmap_name.to_string()), + ..Default::default() + }, + data: Some(BTreeMap::from([("ca.crt".to_string(), ca_pem)])), + ..Default::default() + }; + + api.create(&PostParams::default(), &configmap) + .await + .into_diagnostic() + .wrap_err_with(|| format!("failed to create configmap {configmap_name}"))?; + + info!( + namespace = %namespace, + configmap = %configmap_name, + "Backend CA ConfigMap created." + ); + Ok(()) +} + fn tls_secret(name: &str, crt_pem: &str, key_pem: &str, ca_pem: &str) -> Secret { let mut data = BTreeMap::new(); data.insert( diff --git a/crates/openshell-server/src/cli.rs b/crates/openshell-server/src/cli.rs index c11d895618..9b01b749a4 100644 --- a/crates/openshell-server/src/cli.rs +++ b/crates/openshell-server/src/cli.rs @@ -2314,6 +2314,59 @@ mod tests { .expect("complete package-generated TLS paths may not exist before certificate generation"); } + #[test] + fn generate_certs_backend_ca_configmap_flags_parse() { + let _lock = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let _g1 = EnvVarGuard::remove("OPENSHELL_DB_URL"); + let _g2 = EnvVarGuard::remove("POD_NAMESPACE"); + + let cli = Cli::try_parse_from([ + "openshell-gateway", + "generate-certs", + "--namespace", + "openshell", + "--jwt-only", + "--jwt-secret-name", + "openshell-jwt-keys", + "--backend-ca-configmap-name", + "openshell-backend-ca", + "--backend-ca-source-secret", + "openshell-server-tls", + ]) + .expect("backend CA ConfigMap flags should parse with --jwt-only"); + + assert!(matches!( + cli.command, + Some(super::Commands::GenerateCerts(_)) + )); + } + + #[test] + fn generate_certs_backend_ca_source_secret_requires_configmap_name() { + let _lock = ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let _g1 = EnvVarGuard::remove("OPENSHELL_DB_URL"); + let _g2 = EnvVarGuard::remove("POD_NAMESPACE"); + + let err = Cli::try_parse_from([ + "openshell-gateway", + "generate-certs", + "--namespace", + "openshell", + "--jwt-only", + "--jwt-secret-name", + "openshell-jwt-keys", + "--backend-ca-source-secret", + "openshell-server-tls", + ]) + .expect_err("--backend-ca-source-secret should require --backend-ca-configmap-name"); + + assert_eq!(err.kind(), clap::error::ErrorKind::MissingRequiredArgument); + } + #[test] fn bare_invocation_with_no_db_url_parses_for_runtime_defaults() { // db_url is Option at the clap level so subcommand parsing diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 2b2efde9a2..2bf1c8ae25 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -186,7 +186,7 @@ discovery endpoint or its TLS CA. | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | fullnameOverride | string | `""` | Override the full generated resource name. | -| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of a ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Create this ConfigMap from the CA that signed the server certificate. Required when enabled. | +| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. When pkiInitJob is enabled (default), the certgen hook creates this ConfigMap automatically. With cert-manager, the ConfigMap is created on the first helm upgrade after cert-manager has issued the server certificate. | | grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and a ConfigMap containing the CA certificate for backend validation. | | grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (..svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 98243627dd..f446a11f8e 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -240,6 +240,13 @@ Returns a YAML list. Append extra SANs from values with range loops. | toYaml }} {{- end }} +{{/* +Name of the ConfigMap holding the backend CA for BackendTLSPolicy validation. +*/}} +{{- define "openshell.backendCaConfigMapName" -}} +{{- .Values.grpcRoute.backendTLSPolicy.caCertificateConfigMapName | default (printf "%s-backend-ca" (include "openshell.fullname" .)) -}} +{{- end }} + {{/* Gateway workload kind. StatefulSet is the default because the default SQLite database requires persistent per-pod storage. diff --git a/deploy/helm/openshell/templates/backend-tls-policy.yaml b/deploy/helm/openshell/templates/backend-tls-policy.yaml index 271a604ed9..ead4efbe11 100644 --- a/deploy/helm/openshell/templates/backend-tls-policy.yaml +++ b/deploy/helm/openshell/templates/backend-tls-policy.yaml @@ -2,9 +2,6 @@ # SPDX-License-Identifier: Apache-2.0 {{- if .Values.grpcRoute.backendTLSPolicy.enabled }} -{{- if not .Values.grpcRoute.backendTLSPolicy.caCertificateConfigMapName }} -{{- fail "grpcRoute.backendTLSPolicy.caCertificateConfigMapName is required when grpcRoute.backendTLSPolicy is enabled" }} -{{- end }} {{- if .Values.server.disableTls }} {{- fail "grpcRoute.backendTLSPolicy requires the gateway pod to serve TLS; set server.disableTls=false" }} {{- end }} @@ -24,6 +21,6 @@ spec: caCertificateRefs: - group: "" kind: ConfigMap - name: {{ .Values.grpcRoute.backendTLSPolicy.caCertificateConfigMapName }} + name: {{ include "openshell.backendCaConfigMapName" . }} hostname: {{ default (printf "%s.%s.svc.cluster.local" (include "openshell.fullname" .) .Release.Namespace) .Values.grpcRoute.backendTLSPolicy.hostname }} {{- end }} diff --git a/deploy/helm/openshell/templates/certgen.yaml b/deploy/helm/openshell/templates/certgen.yaml index c4f2d74e85..040ede92dd 100644 --- a/deploy/helm/openshell/templates/certgen.yaml +++ b/deploy/helm/openshell/templates/certgen.yaml @@ -31,6 +31,11 @@ rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"] + {{- if .Values.grpcRoute.backendTLSPolicy.enabled }} + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create"] + {{- end }} --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding @@ -114,4 +119,10 @@ spec: - --server-san={{ . }} {{- end }} {{- end }} + {{- if .Values.grpcRoute.backendTLSPolicy.enabled }} + - --backend-ca-configmap-name={{ include "openshell.backendCaConfigMapName" . }} + {{- if .Values.certManager.enabled }} + - --backend-ca-source-secret={{ .Values.server.tls.certSecretName }} + {{- end }} + {{- end }} {{- end }} diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 5caae49eee..3fb25a4154 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -562,9 +562,12 @@ grpcRoute: # Requires server.disableTls=false and a ConfigMap containing the CA # certificate for backend validation. enabled: false - # -- Name of a ConfigMap containing the CA certificate (key: ca.crt) used - # to validate the gateway pod's TLS certificate. Create this ConfigMap - # from the CA that signed the server certificate. Required when enabled. + # -- Name of the ConfigMap containing the CA certificate (key: ca.crt) used + # to validate the gateway pod's TLS certificate. Defaults to + # -backend-ca when empty. When pkiInitJob is enabled (default), + # the certgen hook creates this ConfigMap automatically. With + # cert-manager, the ConfigMap is created on the first helm upgrade after + # cert-manager has issued the server certificate. caCertificateConfigMapName: "" # -- Hostname the Gateway proxy validates against the backend's TLS # certificate SAN. Defaults to the service FQDN diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index cecd4b1687..7c6a6d02f9 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -147,20 +147,10 @@ This keeps TLS on the gateway pod rather than disabling it with `server.disableT BackendTLSPolicy is a standard Gateway API resource. It is supported on OpenShift 4.22+ (via the OpenShift gateway controller) and on other platforms where the Gateway API implementation supports it (check your controller's documentation). -### Create the backend CA ConfigMap - -After the server certificate is created (by cert-manager or the pkiInitJob), extract the CA certificate into a ConfigMap: - -```shell -kubectl -n openshell get secret openshell-server-tls \ - -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt - -kubectl -n openshell create configmap openshell-backend-ca \ - --from-file=ca.crt=/tmp/ca.crt -``` - ### Install with e2e TLS +The certgen hook automatically creates the backend CA ConfigMap when `backendTLSPolicy` is enabled: + ```shell helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ @@ -173,7 +163,6 @@ helm upgrade --install openshell \ --set grpcRoute.gateway.listener.port=443 \ --set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \ --set grpcRoute.backendTLSPolicy.enabled=true \ - --set grpcRoute.backendTLSPolicy.caCertificateConfigMapName=openshell-backend-ca \ --set server.oidc.issuer=https:// \ --set 'grpcRoute.hostnames[0]=' ``` diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index b14f5a0751..a4c9c264b1 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -158,7 +158,7 @@ The listener TLS Secret should contain the certificate for the external hostname #### Install with e2e TLS -Install the chart with cert-manager, the GRPCRoute, and BackendTLSPolicy enabled. The server certificate SAN list must include the external hostname so the gateway can serve TLS to both the cluster-internal service FQDN and external clients: +Install the chart with the GRPCRoute and BackendTLSPolicy enabled. The certgen hook automatically creates the backend CA ConfigMap from the generated PKI bundle. The server certificate SAN list must include the external hostname so the gateway can serve TLS to both the cluster-internal service FQDN and external clients: ```shell helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ @@ -166,42 +166,25 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ --namespace openshell \ --set podSecurityContext.fsGroup=null \ --set securityContext.runAsUser=null \ - --set certManager.enabled=true \ - --set 'certManager.serverDnsNames[0]=openshell' \ - --set 'certManager.serverDnsNames[1]=openshell.openshell.svc' \ - --set 'certManager.serverDnsNames[2]=openshell.openshell.svc.cluster.local' \ - --set 'certManager.serverDnsNames[3]=localhost' \ - --set 'certManager.serverDnsNames[4]=' \ --set grpcRoute.enabled=true \ --set grpcRoute.gateway.name=openshell-gateway \ --set grpcRoute.gateway.namespace=openshift-ingress \ --set 'grpcRoute.hostnames[0]=' \ --set grpcRoute.backendTLSPolicy.enabled=true \ - --set grpcRoute.backendTLSPolicy.caCertificateConfigMapName=openshell-backend-ca + --set 'pkiInitJob.serverDnsNames[0]=' ``` | Override | Reason | |---|---| | `podSecurityContext.fsGroup=null` / `securityContext.runAsUser=null` | Let OpenShift's SCC admission assign UIDs. | -| `certManager.enabled=true` | Use cert-manager for automatic server certificate lifecycle. | | `grpcRoute.enabled=true` | Create a GRPCRoute pointing at the external Gateway. | | `grpcRoute.gateway.name` / `namespace` | Reference the Gateway created above in `openshift-ingress`. | -| `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. | -| `grpcRoute.backendTLSPolicy.caCertificateConfigMapName` | ConfigMap containing the CA that signed the server certificate. | +| `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. The certgen hook auto-creates the backend CA ConfigMap. | +| `pkiInitJob.serverDnsNames` | Add the external hostname to the server certificate SANs. | Note that `server.disableTls` is **not** set — the gateway pod serves TLS, and the BackendTLSPolicy tells the OpenShift router to connect over TLS. -#### Create the backend CA ConfigMap - -After cert-manager issues the server certificate, create a ConfigMap containing the CA certificate so the BackendTLSPolicy can validate the backend: - -```shell -oc -n openshell get secret openshell-server-tls \ - -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt - -oc -n openshell create configmap openshell-backend-ca \ - --from-file=ca.crt=/tmp/ca.crt -``` +To use cert-manager instead of the built-in PKI, add `--set certManager.enabled=true` and configure `certManager.serverDnsNames` with the external hostname. On first install with cert-manager, the backend CA ConfigMap is created on the first `helm upgrade` after cert-manager issues the server certificate. #### Register over HTTPS From 3d8f1ab9c853600f8de00102b09f5494c586e678 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 14 Aug 2026 12:51:07 -0400 Subject: [PATCH 03/27] refactor(helm): add server.tls.enableMtls flag for mTLS control Replace automatic mTLS disabling based on BackendTLSPolicy with an explicit server.tls.enableMtls flag that defaults to true. The user is now responsible for setting this to false when using BackendTLSPolicy, as ingress proxies cannot present client certificates to backends. Updated: - values.yaml: Added server.tls.enableMtls (default true) - gateway-config.yaml: Check enableMtls instead of backendTLSPolicy - _gateway-workload.tpl: Check enableMtls for client CA mount - Tests: Updated to use enableMtls flag - Docs: Added enableMtls=false to BackendTLSPolicy examples - README: Document new flag and BackendTLSPolicy requirement Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/README.md | 5 +++-- deploy/helm/openshell/templates/_helpers.tpl | 1 + .../openshell/tests/gateway_config_test.yaml | 18 ++++++++++++++++++ .../tests/statefulset_client_ca_test.yaml | 19 +++++++++++++++++++ deploy/helm/openshell/values.yaml | 13 +++++++++---- docs/kubernetes/ingress.mdx | 5 +++-- docs/kubernetes/openshift.mdx | 4 +++- 7 files changed, 56 insertions(+), 9 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 2bf1c8ae25..68cb23aa1e 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -187,7 +187,7 @@ discovery endpoint or its TLS CA. | certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | fullnameOverride | string | `""` | Override the full generated resource name. | | grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. When pkiInitJob is enabled (default), the certgen hook creates this ConfigMap automatically. With cert-manager, the ConfigMap is created on the first helm upgrade after cert-manager has issued the server certificate. | -| grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and a ConfigMap containing the CA certificate for backend validation. | +| grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. | | grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (..svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | @@ -292,8 +292,9 @@ discovery endpoint or its TLS CA. | server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. | | server.telemetryEnabled | bool | `true` | Enable anonymous OpenShell telemetry from the gateway and the sandbox supervisors it launches. | | server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. | -| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). Do not set to null; omit the key to use the default secret name above. | +| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Only used when enableMtls is true. Set to "" to use the default client CA (from pkiInitJob or cert-manager). | | server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. | +| server.tls.enableMtls | bool | `true` | Enable mTLS client certificate authentication. When false, the gateway runs HTTPS-only without requiring client certificates (use OIDC for auth instead). Must be false when using BackendTLSPolicy because ingress proxies cannot present client certificates to the backend. | | server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). | | server.workspaceStorageClass | string | `""` | Kubernetes StorageClass for the workspace PVC in sandbox pods. Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the workspace PVC stays Pending and the sandbox never starts. | | service.healthPort | int | `8081` | Gateway health service port. | diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index f446a11f8e..a35dc96db5 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -104,6 +104,7 @@ defaults. */}} {{- define "openshell.gatewayClientCaEnabled" -}} {{- if .Values.server.disableTls -}} +{{- else if not .Values.server.tls.enableMtls -}} {{- else if eq .Values.server.tls.clientCaSecretName "" -}} {{- else if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) -}} true diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index 9805af8254..2cf7705b8e 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -514,6 +514,24 @@ tests: path: data["gateway.toml"] pattern: '\[openshell\.gateway\.tls\]' + - it: omits client_ca_path from the TLS section when mTLS is disabled + template: templates/gateway-config.yaml + set: + server.tls.enableMtls: false + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.tls\]' + - matchRegex: + path: data["gateway.toml"] + pattern: 'cert_path\s*=' + - matchRegex: + path: data["gateway.toml"] + pattern: 'key_path\s*=' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=' + - it: renders client_ca_path for built-in PKI by default template: templates/gateway-config.yaml asserts: diff --git a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml index 813e47fbe0..df6df15693 100644 --- a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml +++ b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml @@ -64,6 +64,25 @@ tests: - notExists: path: spec.template.spec.volumes[3].secret.items + - it: omits client CA volume and mount when mTLS is disabled + template: templates/statefulset.yaml + set: + pkiInitJob.enabled: true + certManager.enabled: false + server.tls.enableMtls: false + asserts: + - notContains: + path: spec.template.spec.volumes + content: + name: tls-client-ca + any: true + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: tls-client-ca + mountPath: /etc/openshell-tls/client-ca + readOnly: true + # When cert-manager owns TLS, does not share its CA, and no separate client CA # secret is configured, there is no client CA to mount: the volume must not # render rather than mounting an empty secret name. diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 3fb25a4154..d0c69f74d3 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -369,9 +369,14 @@ server: tls: # -- K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. certSecretName: openshell-server-tls + # -- Enable mTLS client certificate authentication. When false, the gateway + # runs HTTPS-only without requiring client certificates (use OIDC for auth + # instead). Must be false when using BackendTLSPolicy because ingress + # proxies cannot present client certificates to the backend. + enableMtls: true # -- K8s secret with ca.crt for client certificate verification (mTLS). - # Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). - # Do not set to null; omit the key to use the default secret name above. + # Only used when enableMtls is true. Set to "" to use the default client CA + # (from pkiInitJob or cert-manager). clientCaSecretName: openshell-server-client-ca # -- K8s secret mounted into sandbox pods for mTLS to the server. clientTlsSecretName: openshell-client-tls @@ -559,8 +564,8 @@ grpcRoute: # -- Create a BackendTLSPolicy resource for end-to-end TLS between the # Gateway proxy and the OpenShell gateway pod. The traffic flow is: # client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. - # Requires server.disableTls=false and a ConfigMap containing the CA - # certificate for backend validation. + # Requires server.disableTls=false and server.tls.enableMtls=false. The + # certgen hook auto-creates the backend CA ConfigMap. enabled: false # -- Name of the ConfigMap containing the CA certificate (key: ca.crt) used # to validate the gateway pod's TLS certificate. Defaults to diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 7c6a6d02f9..028d75ebae 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -143,7 +143,7 @@ As an alternative to the plaintext backend path above, the chart can create a `B client → HTTPS → Gateway (terminate TLS) → TLS (re-encrypt) → openshell gateway pod ``` -This keeps TLS on the gateway pod rather than disabling it with `server.disableTls=true`. The Gateway proxy validates the backend's certificate against a CA you supply in a ConfigMap. +This keeps TLS on the gateway pod rather than disabling it with `server.disableTls=true`. The Gateway proxy validates the backend's certificate against a CA ConfigMap that the certgen hook auto-creates. BackendTLSPolicy is a standard Gateway API resource. It is supported on OpenShift 4.22+ (via the OpenShift gateway controller) and on other platforms where the Gateway API implementation supports it (check your controller's documentation). @@ -156,6 +156,7 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ + --set server.tls.enableMtls=false \ --set grpcRoute.enabled=true \ --set grpcRoute.gateway.create=true \ --set grpcRoute.gateway.className=eg \ @@ -167,7 +168,7 @@ helm upgrade --install openshell \ --set 'grpcRoute.hostnames[0]=' ``` -Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. +Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). For OpenShift 4.22+, see [OpenShift](/kubernetes/openshift#end-to-end-tls-openshift-422) for platform-specific instructions including Gateway and GatewayClass setup. diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index a4c9c264b1..311e79fb6a 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -166,6 +166,7 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ --namespace openshell \ --set podSecurityContext.fsGroup=null \ --set securityContext.runAsUser=null \ + --set server.tls.enableMtls=false \ --set grpcRoute.enabled=true \ --set grpcRoute.gateway.name=openshell-gateway \ --set grpcRoute.gateway.namespace=openshift-ingress \ @@ -177,12 +178,13 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ | Override | Reason | |---|---| | `podSecurityContext.fsGroup=null` / `securityContext.runAsUser=null` | Let OpenShift's SCC admission assign UIDs. | +| `server.tls.enableMtls=false` | Disable mTLS client certificate authentication. BackendTLSPolicy only validates the server certificate; the ingress proxy cannot present a client certificate to the backend. Use OIDC for authentication instead. | | `grpcRoute.enabled=true` | Create a GRPCRoute pointing at the external Gateway. | | `grpcRoute.gateway.name` / `namespace` | Reference the Gateway created above in `openshift-ingress`. | | `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. The certgen hook auto-creates the backend CA ConfigMap. | | `pkiInitJob.serverDnsNames` | Add the external hostname to the server certificate SANs. | -Note that `server.disableTls` is **not** set — the gateway pod serves TLS, and the BackendTLSPolicy tells the OpenShift router to connect over TLS. +Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)). To use cert-manager instead of the built-in PKI, add `--set certManager.enabled=true` and configure `certManager.serverDnsNames` with the external hostname. On first install with cert-manager, the backend CA ConfigMap is created on the first `helm upgrade` after cert-manager issues the server certificate. From 66147ee8c0c3c73e87f8e0dade0b6978bf9ac58a Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Thu, 20 Aug 2026 12:06:28 -0400 Subject: [PATCH 04/27] docs(helm): clarify cert-manager backend CA ConfigMap workflow Update documentation to explain the two-step install process required when using cert-manager with BackendTLSPolicy: 1. helm install - cert-manager issues the server certificate, but the certgen hook can't create the backend CA ConfigMap yet (cert-manager reconciles after templates are applied) 2. helm upgrade - certgen hook reads the CA from the cert-manager-issued certificate and creates the ConfigMap Previously, the docs said "created on first upgrade" without explaining why or that the feature won't work until then. The updated docs now: - Explain the timing issue (cert-manager reconciles after chart install) - Provide clear steps for the cert-manager workflow - Note that pkiInitJob (default) creates it immediately on install - Clarify that users must wait for the Certificate to be Ready before running the second upgrade Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/README.md | 2 +- deploy/helm/openshell/values.yaml | 9 +++++---- docs/kubernetes/ingress.mdx | 2 ++ docs/kubernetes/openshift.mdx | 7 ++++++- 4 files changed, 14 insertions(+), 6 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 68cb23aa1e..b227adb83e 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -186,7 +186,7 @@ discovery endpoint or its TLS CA. | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | fullnameOverride | string | `""` | Override the full generated resource name. | -| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. When pkiInitJob is enabled (default), the certgen hook creates this ConfigMap automatically. With cert-manager, the ConfigMap is created on the first helm upgrade after cert-manager has issued the server certificate. | +| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook waits for cert-manager to issue the server certificate first, so on first install the ConfigMap is created during the subsequent `helm upgrade` (run upgrade after cert-manager reconciles). | | grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. | | grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (..svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index d0c69f74d3..b4ec8709cc 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -569,10 +569,11 @@ grpcRoute: enabled: false # -- Name of the ConfigMap containing the CA certificate (key: ca.crt) used # to validate the gateway pod's TLS certificate. Defaults to - # -backend-ca when empty. When pkiInitJob is enabled (default), - # the certgen hook creates this ConfigMap automatically. With - # cert-manager, the ConfigMap is created on the first helm upgrade after - # cert-manager has issued the server certificate. + # -backend-ca when empty. The certgen hook auto-creates this: + # with pkiInitJob (default), immediately on install/upgrade; with + # cert-manager, the hook waits for cert-manager to issue the server + # certificate first, so on first install the ConfigMap is created during + # the subsequent `helm upgrade` (run upgrade after cert-manager reconciles). caCertificateConfigMapName: "" # -- Hostname the Gateway proxy validates against the backend's TLS # certificate SAN. Defaults to the service FQDN diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 028d75ebae..2b397fc6cf 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -170,6 +170,8 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), you'll need to run `helm upgrade` a second time after cert-manager issues the server certificate so the certgen hook can create the backend CA ConfigMap from it. + For OpenShift 4.22+, see [OpenShift](/kubernetes/openshift#end-to-end-tls-openshift-422) for platform-specific instructions including Gateway and GatewayClass setup. ## SSH Relay diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 311e79fb6a..17576d78a1 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -186,7 +186,12 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)). -To use cert-manager instead of the built-in PKI, add `--set certManager.enabled=true` and configure `certManager.serverDnsNames` with the external hostname. On first install with cert-manager, the backend CA ConfigMap is created on the first `helm upgrade` after cert-manager issues the server certificate. +**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` and configure `certManager.serverDnsNames` with the external hostname. With cert-manager, the backend CA ConfigMap requires a two-step install because cert-manager creates the server certificate *after* the chart is installed: + +1. `helm install` — cert-manager issues the server certificate, but the certgen hook can't create the backend CA ConfigMap yet (logs a warning) +2. `helm upgrade` — certgen hook now reads the CA from the cert-manager-issued server certificate and creates the backend CA ConfigMap + +The gateway won't accept backend connections until step 2 completes. Wait for the cert-manager Certificate to be Ready before running upgrade. #### Register over HTTPS From f1945c93a3833e2eec089ed9f654e6b70ee854df Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Thu, 20 Aug 2026 12:14:04 -0400 Subject: [PATCH 05/27] fix(docs): remove incorrect external hostname requirement for BackendTLSPolicy BackendTLSPolicy validates the backend certificate against the service FQDN (e.g., openshell.openshell.svc.cluster.local), not the external hostname. The external hostname only needs to be on the Gateway listener certificate for client-facing TLS. The default certManager.serverDnsNames already includes all required service FQDN variants, so no configuration is needed for BackendTLSPolicy to work. Fixed incorrect documentation that claimed: - "The server certificate SAN list must include the external hostname" - Users need to "configure certManager.serverDnsNames with the external hostname" Removed the unnecessary pkiInitJob.serverDnsNames override from the example and clarified that: - Gateway listener certificate needs the external hostname (for clients) - Backend certificate needs the service FQDN (for Gateway proxy) - The service FQDN is already in the defaults Signed-off-by: Brandon Squizzato --- docs/kubernetes/openshift.mdx | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 17576d78a1..4b8ccf4a84 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -158,7 +158,7 @@ The listener TLS Secret should contain the certificate for the external hostname #### Install with e2e TLS -Install the chart with the GRPCRoute and BackendTLSPolicy enabled. The certgen hook automatically creates the backend CA ConfigMap from the generated PKI bundle. The server certificate SAN list must include the external hostname so the gateway can serve TLS to both the cluster-internal service FQDN and external clients: +Install the chart with the GRPCRoute and BackendTLSPolicy enabled. The certgen hook automatically creates the backend CA ConfigMap from the generated PKI bundle: ```shell helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ @@ -171,8 +171,7 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ --set grpcRoute.gateway.name=openshell-gateway \ --set grpcRoute.gateway.namespace=openshift-ingress \ --set 'grpcRoute.hostnames[0]=' \ - --set grpcRoute.backendTLSPolicy.enabled=true \ - --set 'pkiInitJob.serverDnsNames[0]=' + --set grpcRoute.backendTLSPolicy.enabled=true ``` | Override | Reason | @@ -181,12 +180,12 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ | `server.tls.enableMtls=false` | Disable mTLS client certificate authentication. BackendTLSPolicy only validates the server certificate; the ingress proxy cannot present a client certificate to the backend. Use OIDC for authentication instead. | | `grpcRoute.enabled=true` | Create a GRPCRoute pointing at the external Gateway. | | `grpcRoute.gateway.name` / `namespace` | Reference the Gateway created above in `openshift-ingress`. | -| `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. The certgen hook auto-creates the backend CA ConfigMap. | -| `pkiInitJob.serverDnsNames` | Add the external hostname to the server certificate SANs. | +| `grpcRoute.backendTLSPolicy.enabled=true` | Create a BackendTLSPolicy for TLS re-encryption to the gateway pod. The certgen hook auto-creates the backend CA ConfigMap. The Gateway proxy validates the backend certificate against the service FQDN, which is already in the default server certificate SANs. | +| `grpcRoute.hostnames` | External hostname for the GRPCRoute. This goes on the Gateway listener certificate, not the backend certificate. | Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)). -**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` and configure `certManager.serverDnsNames` with the external hostname. With cert-manager, the backend CA ConfigMap requires a two-step install because cert-manager creates the server certificate *after* the chart is installed: +**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. With cert-manager, the backend CA ConfigMap requires a two-step install because cert-manager creates the server certificate *after* the chart is installed: 1. `helm install` — cert-manager issues the server certificate, but the certgen hook can't create the backend CA ConfigMap yet (logs a warning) 2. `helm upgrade` — certgen hook now reads the CA from the cert-manager-issued server certificate and creates the backend CA ConfigMap From be242646cee5787c5b1982c4e69f1b9206e58f84 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Thu, 20 Aug 2026 13:07:57 -0400 Subject: [PATCH 06/27] docs: clarify ACME with LetsEncrypt reference Change all references from "ACME issuer" to "LetsEncrypt/ACME issuer" to help users understand that LetsEncrypt is the most common ACME provider and what ACME means in practice. Updated: - docs/kubernetes/managing-certificates.mdx - docs/kubernetes/openshift.mdx - deploy/helm/openshell/values.yaml - deploy/helm/openshell/README.md - deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/README.md | 2 +- .../helm/openshell/ci/values-openshift-route-cert-manager.yaml | 2 +- deploy/helm/openshell/values.yaml | 2 +- docs/kubernetes/managing-certificates.mdx | 2 +- docs/kubernetes/openshift.mdx | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index b227adb83e..31bb4ccdc8 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -184,7 +184,7 @@ discovery endpoint or its TLS CA. | certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. | | certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. | | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | -| certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | +| certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | fullnameOverride | string | `""` | Override the full generated resource name. | | grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook waits for cert-manager to issue the server certificate first, so on first install the ConfigMap is created during the subsequent `helm upgrade` (run upgrade after cert-manager reconciles). | | grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. | diff --git a/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml b/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml index e434e98b8e..de44871361 100644 --- a/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml +++ b/deploy/helm/openshell/ci/values-openshift-route-cert-manager.yaml @@ -2,7 +2,7 @@ # SPDX-License-Identifier: Apache-2.0 # Render-coverage overlay for cert-manager issuing the server certificate from -# an external Issuer/ClusterIssuer (e.g. a real ACME issuer), plus an +# an external Issuer/ClusterIssuer (e.g. a real LetsEncrypt/ACME issuer), plus an # OpenShift Route with TLS passthrough. Merge after values.yaml: # helm lint deploy/helm/openshell -f ci/values-openshift-route-cert-manager.yaml # diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index b4ec8709cc..95b778b22a 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -479,7 +479,7 @@ certManager: # -- Secret created for the intermediate CA (Certificate with isCA: true). caSecretName: openshell-ca-tls # -- Override the issuerRef for the external server Certificate (e.g. a real - # ACME ClusterIssuer for a publicly-trusted cert on an external hostname). + # LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). # When set, the chart creates a second server certificate from this issuer # with only the hostnames in serverDnsNames; the internal server certificate # is always signed by the chart's own CA. Leave name empty to use the chart diff --git a/docs/kubernetes/managing-certificates.mdx b/docs/kubernetes/managing-certificates.mdx index c4cb07f57e..d97c1e24be 100644 --- a/docs/kubernetes/managing-certificates.mdx +++ b/docs/kubernetes/managing-certificates.mdx @@ -66,7 +66,7 @@ By default, cert-manager issues both the server and client certificates from a self-signed CA the chart creates — this rotates automatically, but the server certificate is still not publicly trusted. `certManager.serverIssuerRef` overrides the `issuerRef` on the server `Certificate` resource to point at a -real `Issuer` or `ClusterIssuer` instead, for example an ACME issuer: +real `Issuer` or `ClusterIssuer` instead, for example a LetsEncrypt/ACME issuer: ```shell helm upgrade --install openshell \ diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 4b8ccf4a84..b37a625e9c 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -201,7 +201,7 @@ openshell status ### Route with TLS Passthrough (All OpenShift Versions) -For OpenShift versions prior to 4.22, or when you prefer Route-based ingress, cert-manager can issue the gateway's server certificate from a real Issuer or ClusterIssuer (for example, an ACME issuer), and an OpenShift Route with TLS passthrough exposes it externally while the gateway keeps terminating its own TLS and mTLS. +For OpenShift versions prior to 4.22, or when you prefer Route-based ingress, cert-manager can issue the gateway's server certificate from a real Issuer or ClusterIssuer (for example, a LetsEncrypt/ACME issuer), and an OpenShift Route with TLS passthrough exposes it externally while the gateway keeps terminating its own TLS and mTLS. Install cert-manager and configure a working `ClusterIssuer` first — see [Managing Certificates](/kubernetes/managing-certificates) for the From 77ce10bddd2d7a248398a5fec1ec9fc7a99235c6 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Thu, 20 Aug 2026 13:11:02 -0400 Subject: [PATCH 07/27] docs(openshift): restructure end-to-end TLS options and clarify Gateway hostname Reorganize the OpenShift production deployment documentation: 1. Changed main section from "Production Deployments" to "Options for end-to-end TLS" for better clarity 2. Renamed subsections for consistency and clarity: - "End-to-end TLS using Gateway API and BackendTLSPolicy (OpenShift 4.22+)" - "End-to-end TLS using pass-through Route (all OpenShift versions)" 3. Clarified that the Gateway hostname is typically a wildcard: "typically a wildcard like *.openshell-ingress-gw.example.com" 4. Removed the recommendation to copy the cluster's wildcard certificate from openshift-ingress namespace, as this is not a recommended security best practice These changes make it clearer that users have two end-to-end TLS options and help them understand the typical naming pattern for Gateway hostnames. Signed-off-by: Brandon Squizzato --- docs/kubernetes/openshift.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index b37a625e9c..d38347c8a7 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -87,11 +87,11 @@ openshell gateway add http://127.0.0.1:8080 --local --name openshift openshell status ``` -## Production Deployments +## Options for end-to-end TLS -The steps above run the gateway over plaintext HTTP for quick evaluation. For production deployments, choose one of the approaches below based on your OpenShift version. +The steps above run the gateway over plaintext HTTP for quick evaluation. For production deployments, choose one of the approaches below based on your OpenShift version and preferences. -### End-to-end TLS (OpenShift 4.22+) +### End-to-end TLS using Gateway API and BackendTLSPolicy (OpenShift 4.22+) OpenShift 4.22 and later support `BackendTLSPolicy` in the Gateway API, enabling end-to-end TLS between the OpenShift router and the OpenShell gateway pod. The traffic flow is: @@ -124,7 +124,7 @@ EOF #### Create the Gateway -Create a Gateway resource in the `openshift-ingress` namespace. Replace `` with your cluster's route hostname: +Create a Gateway resource in the `openshift-ingress` namespace. Replace `` with your cluster's route hostname (typically a wildcard like `*.openshell-ingress-gw.example.com`): ```shell oc apply -f - <<'EOF' @@ -154,7 +154,7 @@ spec: EOF ``` -The listener TLS Secret should contain the certificate for the external hostname. On ROSA and managed OpenShift clusters, you can copy the cluster's wildcard certificate from the `openshift-ingress` namespace. +The listener TLS Secret should contain the certificate for the external hostname. #### Install with e2e TLS @@ -199,7 +199,7 @@ openshell gateway add https:// --name openshift openshell status ``` -### Route with TLS Passthrough (All OpenShift Versions) +### End-to-end TLS using pass-through Route (all OpenShift versions) For OpenShift versions prior to 4.22, or when you prefer Route-based ingress, cert-manager can issue the gateway's server certificate from a real Issuer or ClusterIssuer (for example, a LetsEncrypt/ACME issuer), and an OpenShift Route with TLS passthrough exposes it externally while the gateway keeps terminating its own TLS and mTLS. From 5dfc3883ed3eba433dfe2ba7bc733880bda021e4 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 14:45:08 -0400 Subject: [PATCH 08/27] feat(helm): eliminate two-stage install for BackendTLSPolicy with cert-manager When using BackendTLSPolicy with cert-manager, the certgen hook now polls for up to 90 seconds waiting for cert-manager to issue the TLS certificate before creating the backend CA ConfigMap. This eliminates the need for a second `helm upgrade` in most cases. The hook polls every 2 seconds with progress logging every 10 seconds. If cert-manager takes longer than 90 seconds, the hook times out gracefully and logs a warning, preserving the fallback to manual ConfigMap creation or a second upgrade. The Job's activeDeadlineSeconds is 120s, so the 90s timeout leaves 30s margin for ConfigMap creation and hook completion. Signed-off-by: Brandon Squizzato --- crates/openshell-server/src/certgen.rs | 91 +++++++++++++++++--------- deploy/helm/openshell/values.yaml | 7 +- docs/kubernetes/ingress.mdx | 2 +- 3 files changed, 65 insertions(+), 35 deletions(-) diff --git a/crates/openshell-server/src/certgen.rs b/crates/openshell-server/src/certgen.rs index 01c920242a..d54caafb83 100644 --- a/crates/openshell-server/src/certgen.rs +++ b/crates/openshell-server/src/certgen.rs @@ -79,16 +79,16 @@ pub struct CertgenArgs { #[arg(long)] dry_run: bool, - /// Name of a ConfigMap to create containing the CA certificate (key: ca.crt) - /// for BackendTLSPolicy backend validation. In full PKI mode, the CA comes + /// Name of a `ConfigMap` to create containing the CA certificate (key: ca.crt) + /// for `BackendTLSPolicy` backend validation. In full PKI mode, the CA comes /// from the generated bundle. In --jwt-only mode, the CA is read from /// --backend-ca-source-secret. #[arg(long, value_name = "NAME")] backend_ca_configmap_name: Option, - /// Name of an existing Secret containing a ca.crt key to populate the - /// backend CA ConfigMap from. Required with --jwt-only when - /// --backend-ca-configmap-name is set (typically the server TLS Secret + /// Name of an existing `Secret` containing a ca.crt key to populate the + /// backend CA `ConfigMap` from. Required with --jwt-only when + /// --backend-ca-configmap-name is set (typically the server TLS `Secret` /// created by cert-manager). #[arg(long, value_name = "NAME", requires = "backend_ca_configmap_name")] backend_ca_source_secret: Option, @@ -348,33 +348,62 @@ async fn create_backend_ca_configmap_if_needed( bundle.ca_cert_pem.clone() } else if let Some(source_secret) = &args.backend_ca_source_secret { let secret_api: Api = Api::namespaced(client, namespace); - match secret_api - .get_opt(source_secret) - .await - .into_diagnostic() - .wrap_err_with(|| format!("failed to read secret {source_secret}"))? - { - Some(secret) => { - let data = secret.data.ok_or_else(|| { - miette::miette!("secret {source_secret} has no data") - })?; - let ca = data.get("ca.crt").ok_or_else(|| { - miette::miette!("secret {source_secret} has no ca.crt key") - })?; - String::from_utf8(ca.0.clone()) - .into_diagnostic() - .wrap_err("ca.crt is not valid UTF-8")? - } - None => { - warn!( - secret = %source_secret, - configmap = %configmap_name, - "Backend CA source secret not found; ConfigMap not created. \ - Create it manually or run helm upgrade after the TLS secret exists." - ); - return Ok(()); + + // Poll for the cert-manager Secret with timeout to handle cert issuance delay. + // The Job has activeDeadlineSeconds: 120, so we poll for up to 90s leaving + // margin for ConfigMap creation and hook completion. + let poll_timeout = std::time::Duration::from_secs(90); + let poll_interval = std::time::Duration::from_secs(2); + let start = std::time::Instant::now(); + + let secret = loop { + match secret_api + .get_opt(source_secret) + .await + .into_diagnostic() + .wrap_err_with(|| format!("failed to read secret {source_secret}"))? + { + Some(secret) => break secret, + None if start.elapsed() >= poll_timeout => { + warn!( + secret = %source_secret, + configmap = %configmap_name, + timeout_secs = poll_timeout.as_secs(), + "Backend CA source secret not found after polling; ConfigMap not created. \ + This is expected if cert-manager is still issuing the certificate. \ + Run helm upgrade after the TLS secret exists or the BackendTLSPolicy \ + will remain non-functional until the ConfigMap is created manually." + ); + return Ok(()); + } + None => { + if start.elapsed().as_secs().is_multiple_of(10) { + info!( + secret = %source_secret, + elapsed_secs = start.elapsed().as_secs(), + "Waiting for cert-manager to issue TLS certificate..." + ); + } + tokio::time::sleep(poll_interval).await; + } } - } + }; + + info!( + secret = %source_secret, + elapsed_secs = start.elapsed().as_secs(), + "cert-manager TLS certificate found." + ); + + let data = secret + .data + .ok_or_else(|| miette::miette!("secret {source_secret} has no data"))?; + let ca = data + .get("ca.crt") + .ok_or_else(|| miette::miette!("secret {source_secret} has no ca.crt key"))?; + String::from_utf8(ca.0.clone()) + .into_diagnostic() + .wrap_err("ca.crt is not valid UTF-8")? } else { return Err(miette::miette!( "--backend-ca-source-secret is required with --jwt-only \ diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 95b778b22a..39fdad7bb7 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -571,9 +571,10 @@ grpcRoute: # to validate the gateway pod's TLS certificate. Defaults to # -backend-ca when empty. The certgen hook auto-creates this: # with pkiInitJob (default), immediately on install/upgrade; with - # cert-manager, the hook waits for cert-manager to issue the server - # certificate first, so on first install the ConfigMap is created during - # the subsequent `helm upgrade` (run upgrade after cert-manager reconciles). + # cert-manager, the hook polls for up to 90 seconds waiting for cert-manager + # to issue the server certificate, then creates the ConfigMap. A single + # install usually succeeds; if cert-manager takes longer than 90 seconds, + # run `helm upgrade` after the certificate is issued. caCertificateConfigMapName: "" # -- Hostname the Gateway proxy validates against the backend's TLS # certificate SAN. Defaults to the service FQDN diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 2b397fc6cf..84849968aa 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -170,7 +170,7 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), you'll need to run `helm upgrade` a second time after cert-manager issues the server certificate so the certgen hook can create the backend CA ConfigMap from it. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the certgen hook polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds, the hook times out gracefully and you'll need to run `helm upgrade` after the certificate is issued to create the backend CA ConfigMap. For OpenShift 4.22+, see [OpenShift](/kubernetes/openshift#end-to-end-tls-openshift-422) for platform-specific instructions including Gateway and GatewayClass setup. From 07c297a88be03fd96f29490c7b0dc37ed67845da Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 15:24:40 -0400 Subject: [PATCH 09/27] feat(helm): add configurable timeout for certgen hook Add `pkiInitJob.timeoutSeconds` Helm value (default 120) to control how long the certgen hook Job can run. When using cert-manager with BackendTLSPolicy, the hook polls for (timeoutSeconds - 30) seconds to leave margin for ConfigMap creation and cleanup. This allows users to increase the timeout for environments where cert-manager takes longer than 90 seconds to issue certificates, without requiring code changes. Example usage: ```yaml pkiInitJob: timeoutSeconds: 180 # Hook polls for 150 seconds ``` Signed-off-by: Brandon Squizzato --- crates/openshell-server/src/certgen.rs | 10 ++++++++-- deploy/helm/openshell/templates/certgen.yaml | 3 ++- deploy/helm/openshell/values.yaml | 5 +++++ 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/crates/openshell-server/src/certgen.rs b/crates/openshell-server/src/certgen.rs index d54caafb83..b31f10dd32 100644 --- a/crates/openshell-server/src/certgen.rs +++ b/crates/openshell-server/src/certgen.rs @@ -92,6 +92,12 @@ pub struct CertgenArgs { /// created by cert-manager). #[arg(long, value_name = "NAME", requires = "backend_ca_configmap_name")] backend_ca_source_secret: Option, + + /// Maximum time in seconds to poll for the backend CA source `Secret` when + /// using cert-manager. Defaults to 90 seconds. The Helm chart sets this to + /// (Job activeDeadlineSeconds - 30) to leave margin for `ConfigMap` creation. + #[arg(long, value_name = "SECONDS", default_value = "90")] + backend_ca_poll_timeout_seconds: u64, } pub async fn run(args: CertgenArgs) -> Result<()> { @@ -350,9 +356,9 @@ async fn create_backend_ca_configmap_if_needed( let secret_api: Api = Api::namespaced(client, namespace); // Poll for the cert-manager Secret with timeout to handle cert issuance delay. - // The Job has activeDeadlineSeconds: 120, so we poll for up to 90s leaving + // The Helm chart sets this to (Job activeDeadlineSeconds - 30) leaving // margin for ConfigMap creation and hook completion. - let poll_timeout = std::time::Duration::from_secs(90); + let poll_timeout = std::time::Duration::from_secs(args.backend_ca_poll_timeout_seconds); let poll_interval = std::time::Duration::from_secs(2); let start = std::time::Instant::now(); diff --git a/deploy/helm/openshell/templates/certgen.yaml b/deploy/helm/openshell/templates/certgen.yaml index 040ede92dd..6d0ce70bb3 100644 --- a/deploy/helm/openshell/templates/certgen.yaml +++ b/deploy/helm/openshell/templates/certgen.yaml @@ -70,7 +70,7 @@ metadata: helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: backoffLimit: 3 - activeDeadlineSeconds: 120 + activeDeadlineSeconds: {{ .Values.pkiInitJob.timeoutSeconds }} ttlSecondsAfterFinished: 300 template: metadata: @@ -123,6 +123,7 @@ spec: - --backend-ca-configmap-name={{ include "openshell.backendCaConfigMapName" . }} {{- if .Values.certManager.enabled }} - --backend-ca-source-secret={{ .Values.server.tls.certSecretName }} + - --backend-ca-poll-timeout-seconds={{ sub .Values.pkiInitJob.timeoutSeconds 30 }} {{- end }} {{- end }} {{- end }} diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 39fdad7bb7..05ce7375d2 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -468,6 +468,11 @@ pkiInitJob: serverDnsNames: [] # -- Extra IP SANs to append to the server certificate. serverIpAddresses: [] + # -- Maximum time in seconds for the certgen hook Job to run. When using + # cert-manager with BackendTLSPolicy, the hook polls for the certificate + # for (timeoutSeconds - 30) to leave margin for ConfigMap creation. Increase + # this if cert-manager takes longer than 90 seconds to issue certificates. + timeoutSeconds: 120 # cert-manager Certificate/Issuer resources (requires cert-manager CRDs in-cluster). # Does not install cert-manager itself. From 62d9b6b4956df98061749143444120fac59c0e94 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 15:25:09 -0400 Subject: [PATCH 10/27] docs(helm): document configurable certgen timeout Update documentation to mention the pkiInitJob.timeoutSeconds value and how it affects the cert-manager polling behavior when using BackendTLSPolicy. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/values.yaml | 9 +++++---- docs/kubernetes/ingress.mdx | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 05ce7375d2..b54b7875cf 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -576,10 +576,11 @@ grpcRoute: # to validate the gateway pod's TLS certificate. Defaults to # -backend-ca when empty. The certgen hook auto-creates this: # with pkiInitJob (default), immediately on install/upgrade; with - # cert-manager, the hook polls for up to 90 seconds waiting for cert-manager - # to issue the server certificate, then creates the ConfigMap. A single - # install usually succeeds; if cert-manager takes longer than 90 seconds, - # run `helm upgrade` after the certificate is issued. + # cert-manager, the hook polls for (pkiInitJob.timeoutSeconds - 30) seconds + # waiting for cert-manager to issue the server certificate, then creates the + # ConfigMap. A single install usually succeeds; if cert-manager takes longer, + # increase pkiInitJob.timeoutSeconds or run `helm upgrade` after the + # certificate is issued. caCertificateConfigMapName: "" # -- Hostname the Gateway proxy validates against the backend's TLS # certificate SAN. Defaults to the service FQDN diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 84849968aa..0d51eb04be 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -170,7 +170,7 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the certgen hook polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds, the hook times out gracefully and you'll need to run `helm upgrade` after the certificate is issued to create the backend CA ConfigMap. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the certgen hook polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value), or run `helm upgrade` after the certificate is issued to create the backend CA ConfigMap. For OpenShift 4.22+, see [OpenShift](/kubernetes/openshift#end-to-end-tls-openshift-422) for platform-specific instructions including Gateway and GatewayClass setup. From 38b5097a0678aa517435192bc135005b9298bc5f Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 15:41:03 -0400 Subject: [PATCH 11/27] feat(helm): add configurable failure behavior for certgen timeout Add `pkiInitJob.failOnTimeout` Helm value (default false) to control whether the certgen hook fails or succeeds when cert-manager does not issue a certificate within the polling timeout. When false (default), the hook succeeds with a warning and users can run `helm upgrade` after cert-manager issues the certificate to create the backend CA ConfigMap. This provides backwards-compatible behavior. When true, the hook fails immediately if the timeout is reached, providing clear feedback that BackendTLSPolicy is non-functional. This is useful for strict validation requirements where incomplete installs should fail fast. Example usage: ```yaml pkiInitJob: timeoutSeconds: 180 failOnTimeout: true # Fail install if cert-manager takes >150s ``` Signed-off-by: Brandon Squizzato --- crates/openshell-server/src/certgen.rs | 22 +++++++++++++++++--- deploy/helm/openshell/templates/certgen.yaml | 3 +++ deploy/helm/openshell/values.yaml | 7 +++++++ docs/kubernetes/ingress.mdx | 2 +- 4 files changed, 30 insertions(+), 4 deletions(-) diff --git a/crates/openshell-server/src/certgen.rs b/crates/openshell-server/src/certgen.rs index b31f10dd32..b42de5c422 100644 --- a/crates/openshell-server/src/certgen.rs +++ b/crates/openshell-server/src/certgen.rs @@ -98,6 +98,13 @@ pub struct CertgenArgs { /// (Job activeDeadlineSeconds - 30) to leave margin for `ConfigMap` creation. #[arg(long, value_name = "SECONDS", default_value = "90")] backend_ca_poll_timeout_seconds: u64, + + /// Fail with an error if the backend CA source `Secret` is not found within + /// the polling timeout. When false (default), the hook succeeds with a warning + /// and the `ConfigMap` is not created. The Helm chart sets this based on + /// pkiInitJob.failOnTimeout. + #[arg(long)] + backend_ca_fail_on_timeout: bool, } pub async fn run(args: CertgenArgs) -> Result<()> { @@ -371,13 +378,22 @@ async fn create_backend_ca_configmap_if_needed( { Some(secret) => break secret, None if start.elapsed() >= poll_timeout => { + let msg = format!( + "Backend CA source secret {source_secret} not found after {timeout_secs}s; \ + ConfigMap {configmap_name} not created. This is expected if cert-manager \ + is still issuing the certificate.", + timeout_secs = poll_timeout.as_secs() + ); + if args.backend_ca_fail_on_timeout { + return Err(miette::miette!( + "{msg} Install failed due to --backend-ca-fail-on-timeout." + )); + } warn!( secret = %source_secret, configmap = %configmap_name, timeout_secs = poll_timeout.as_secs(), - "Backend CA source secret not found after polling; ConfigMap not created. \ - This is expected if cert-manager is still issuing the certificate. \ - Run helm upgrade after the TLS secret exists or the BackendTLSPolicy \ + "{msg} Run helm upgrade after the TLS secret exists or the BackendTLSPolicy \ will remain non-functional until the ConfigMap is created manually." ); return Ok(()); diff --git a/deploy/helm/openshell/templates/certgen.yaml b/deploy/helm/openshell/templates/certgen.yaml index 6d0ce70bb3..138c4d9cd2 100644 --- a/deploy/helm/openshell/templates/certgen.yaml +++ b/deploy/helm/openshell/templates/certgen.yaml @@ -124,6 +124,9 @@ spec: {{- if .Values.certManager.enabled }} - --backend-ca-source-secret={{ .Values.server.tls.certSecretName }} - --backend-ca-poll-timeout-seconds={{ sub .Values.pkiInitJob.timeoutSeconds 30 }} + {{- if .Values.pkiInitJob.failOnTimeout }} + - --backend-ca-fail-on-timeout + {{- end }} {{- end }} {{- end }} {{- end }} diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index b54b7875cf..1ef4174741 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -473,6 +473,13 @@ pkiInitJob: # for (timeoutSeconds - 30) to leave margin for ConfigMap creation. Increase # this if cert-manager takes longer than 90 seconds to issue certificates. timeoutSeconds: 120 + # -- Fail the helm install/upgrade if cert-manager does not issue the + # certificate within the polling timeout. When false (default), the hook + # succeeds with a warning and you can run `helm upgrade` after cert-manager + # issues the certificate to create the backend CA ConfigMap. When true, the + # install fails immediately if the timeout is reached, providing clear feedback + # that BackendTLSPolicy is non-functional. + failOnTimeout: false # cert-manager Certificate/Issuer resources (requires cert-manager CRDs in-cluster). # Does not install cert-manager itself. diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 0d51eb04be..9bca1b0acb 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -170,7 +170,7 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the certgen hook polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value), or run `helm upgrade` after the certificate is issued to create the backend CA ConfigMap. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the certgen hook polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value). By default, the hook succeeds with a warning if the timeout is reached; set `--set pkiInitJob.failOnTimeout=true` to make the install fail immediately instead, providing clear feedback that the BackendTLSPolicy is non-functional. For OpenShift 4.22+, see [OpenShift](/kubernetes/openshift#end-to-end-tls-openshift-422) for platform-specific instructions including Gateway and GatewayClass setup. From 963138e53efa53f5ee1e2dbb679978afea8f7be4 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 15:43:14 -0400 Subject: [PATCH 12/27] feat(helm): change failOnTimeout default to true and add troubleshooting docs Change `pkiInitJob.failOnTimeout` default from false to true to provide immediate feedback when cert-manager does not issue certificates within the polling timeout. This prevents silent failures where BackendTLSPolicy is non-functional but the install appears to succeed. Add comprehensive troubleshooting section to docs/kubernetes/ingress.mdx documenting the specific error "TLS error: Secret is not supplied by SDS" that occurs when the backend CA ConfigMap is missing, with step-by-step resolution instructions. Updated comments in values.yaml to clearly document the default behavior and explain when administrators might see connectivity errors if they override the default to failOnTimeout=false. BREAKING CHANGE: pkiInitJob.failOnTimeout now defaults to true. Helm installs will fail if cert-manager takes longer than (timeoutSeconds - 30) seconds to issue certificates. To restore the old behavior of allowing installs to succeed with a warning, set `pkiInitJob.failOnTimeout=false`. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/values.yaml | 19 +++++++++++-------- docs/kubernetes/ingress.mdx | 24 +++++++++++++++++++++++- 2 files changed, 34 insertions(+), 9 deletions(-) diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 1ef4174741..068591ee45 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -474,12 +474,14 @@ pkiInitJob: # this if cert-manager takes longer than 90 seconds to issue certificates. timeoutSeconds: 120 # -- Fail the helm install/upgrade if cert-manager does not issue the - # certificate within the polling timeout. When false (default), the hook - # succeeds with a warning and you can run `helm upgrade` after cert-manager - # issues the certificate to create the backend CA ConfigMap. When true, the - # install fails immediately if the timeout is reached, providing clear feedback - # that BackendTLSPolicy is non-functional. - failOnTimeout: false + # certificate within the polling timeout. When true (default), the install + # fails immediately if the timeout is reached, providing clear feedback that + # BackendTLSPolicy is non-functional. When false, the hook succeeds with a + # warning and you can run `helm upgrade` after cert-manager issues the + # certificate to create the backend CA ConfigMap. If you set this to false and + # see "TLS error: Secret is not supplied by SDS" when connecting to the gateway, + # check if the TLS secret exists and run `helm upgrade` to create the ConfigMap. + failOnTimeout: true # cert-manager Certificate/Issuer resources (requires cert-manager CRDs in-cluster). # Does not install cert-manager itself. @@ -586,8 +588,9 @@ grpcRoute: # cert-manager, the hook polls for (pkiInitJob.timeoutSeconds - 30) seconds # waiting for cert-manager to issue the server certificate, then creates the # ConfigMap. A single install usually succeeds; if cert-manager takes longer, - # increase pkiInitJob.timeoutSeconds or run `helm upgrade` after the - # certificate is issued. + # increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), + # the install fails if the timeout is reached; set failOnTimeout=false to allow + # the install to succeed and run `helm upgrade` after the certificate is issued. caCertificateConfigMapName: "" # -- Hostname the Gateway proxy validates against the backend's TLS # certificate SAN. Defaults to the service FQDN diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 9bca1b0acb..c29578934c 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -170,7 +170,29 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the certgen hook polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value). By default, the hook succeeds with a warning if the timeout is reached; set `--set pkiInitJob.failOnTimeout=true` to make the install fail immediately instead, providing clear feedback that the BackendTLSPolicy is non-functional. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the certgen hook polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value). By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. + +### Troubleshooting + +If you see the error `remote connection failure, transport failure reason: TLS error: Secret is not supplied by SDS` when connecting through the Gateway: + +1. Check if the backend CA ConfigMap exists: + ```shell + kubectl get configmap -backend-ca -n + ``` + +2. If the ConfigMap is missing, verify the TLS secret exists: + ```shell + kubectl get secret -server-tls -n + ``` + +3. If the secret exists but the ConfigMap doesn't, run `helm upgrade` to create it: + ```shell + helm upgrade oci://ghcr.io/nvidia/openshell/helm-chart \ + --reuse-values --namespace + ``` + +This situation can occur if you set `pkiInitJob.failOnTimeout=false` and cert-manager issued the certificate after the hook timed out. For OpenShift 4.22+, see [OpenShift](/kubernetes/openshift#end-to-end-tls-openshift-422) for platform-specific instructions including Gateway and GatewayClass setup. From e1156e4002fe7ae82fc843608b910f4fac633f65 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 15:58:16 -0400 Subject: [PATCH 13/27] fix(helm): make cert-manager resources pre-install hooks to fix ordering Make Certificate and Issuer resources run as pre-install/pre-upgrade hooks with weight -30, before the certgen hook (weight -20). This fixes the chicken-and-egg problem where the certgen hook was waiting for Secrets created by Certificates that hadn't been created yet. **Hook ordering:** 1. Certificate and Issuer resources created (weight -30) 2. cert-manager issues certificates and creates Secrets 3. certgen hook runs (weight -20), finds Secrets, creates ConfigMap 4. Main resources (StatefulSet, Service, etc.) created Previously, the certgen pre-install hook would run before any resources were created, poll for a non-existent Secret, timeout, and fail. The Certificate resources would never get created because Helm waits for all pre-install hooks to succeed before creating main resources. This fix allows single-stage installs to work reliably as long as cert-manager can issue certificates within the polling timeout. Signed-off-by: Brandon Squizzato --- .../openshell/templates/cert-manager-pki.yaml | 18 ++++++++++++++++++ docs/kubernetes/ingress.mdx | 2 +- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/deploy/helm/openshell/templates/cert-manager-pki.yaml b/deploy/helm/openshell/templates/cert-manager-pki.yaml index 838534e62e..54b4bfef5d 100644 --- a/deploy/helm/openshell/templates/cert-manager-pki.yaml +++ b/deploy/helm/openshell/templates/cert-manager-pki.yaml @@ -8,6 +8,9 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" spec: selfSigned: {} --- @@ -18,6 +21,9 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" spec: isCA: true commonName: openshell-ca @@ -37,6 +43,9 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" spec: ca: secretName: {{ .Values.certManager.caSecretName | quote }} @@ -66,6 +75,9 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" spec: secretName: {{ .Values.server.tls.certSecretName | quote }} duration: {{ .Values.certManager.certificateDuration | quote }} @@ -107,6 +119,9 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" spec: secretName: {{ include "openshell.fullname" . }}-server-external-tls duration: {{ .Values.certManager.certificateDuration | quote }} @@ -138,6 +153,9 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: pre-install,pre-upgrade + helm.sh/hook-weight: "-30" spec: secretName: {{ .Values.server.tls.clientTlsSecretName | quote }} duration: {{ .Values.certManager.certificateDuration | quote }} diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index c29578934c..4eac2bf334 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -170,7 +170,7 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the certgen hook polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value). By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds to issue certificates, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value). By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. ### Troubleshooting From 669159847d45fbd322a415fc31d6e847b5e7c91c Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 16:12:01 -0400 Subject: [PATCH 14/27] feat(helm): add validation to prevent enableMtls with BackendTLSPolicy Add Helm chart validation that fails the install if both server.tls.enableMtls=true and grpcRoute.backendTLSPolicy.enabled=true are set, since this is an invalid configuration. BackendTLSPolicy requires mTLS to be disabled because the Gateway proxy cannot present client certificates to the backend. This validation provides immediate, clear feedback at install time rather than allowing the misconfiguration to be discovered through runtime errors. Example error message: ``` Error: grpcRoute.backendTLSPolicy requires mTLS to be disabled because the Gateway proxy cannot present client certificates to the backend; set server.tls.enableMtls=false ``` Also updated documentation to mention this validation check. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/templates/backend-tls-policy.yaml | 3 +++ docs/kubernetes/ingress.mdx | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/deploy/helm/openshell/templates/backend-tls-policy.yaml b/deploy/helm/openshell/templates/backend-tls-policy.yaml index ead4efbe11..8d7a4fd557 100644 --- a/deploy/helm/openshell/templates/backend-tls-policy.yaml +++ b/deploy/helm/openshell/templates/backend-tls-policy.yaml @@ -5,6 +5,9 @@ {{- if .Values.server.disableTls }} {{- fail "grpcRoute.backendTLSPolicy requires the gateway pod to serve TLS; set server.disableTls=false" }} {{- end }} +{{- if .Values.server.tls.enableMtls }} +{{- fail "grpcRoute.backendTLSPolicy requires mTLS to be disabled because the Gateway proxy cannot present client certificates to the backend; set server.tls.enableMtls=false" }} +{{- end }} apiVersion: gateway.networking.k8s.io/v1 kind: BackendTLSPolicy metadata: diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 4eac2bf334..2e443e0b4a 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -168,7 +168,7 @@ helm upgrade --install openshell \ --set 'grpcRoute.hostnames[0]=' ``` -Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). +Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The chart will fail the install if you try to enable both `grpcRoute.backendTLSPolicy.enabled=true` and `server.tls.enableMtls=true` simultaneously. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds to issue certificates, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value). By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. From cd2cf6f9a90ea8e485c226475b2cf1fbdc85c55f Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 16:14:25 -0400 Subject: [PATCH 15/27] docs(helm): clarify pkiInitJob.timeoutSeconds polling behavior Improve documentation to clearly explain that pkiInitJob.timeoutSeconds controls the Job deadline, but the actual polling timeout is (timeoutSeconds - 30) to reserve 30 seconds for ConfigMap creation and cleanup. Added concrete example: "timeoutSeconds=180 allows 150 seconds of polling" to make the relationship explicit and avoid confusion where users might expect the hook to poll for the full timeout value. Updated both values.yaml inline comments and ingress.mdx documentation for consistency. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/values.yaml | 6 ++++-- docs/kubernetes/ingress.mdx | 4 +++- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 068591ee45..074d099c34 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -470,8 +470,10 @@ pkiInitJob: serverIpAddresses: [] # -- Maximum time in seconds for the certgen hook Job to run. When using # cert-manager with BackendTLSPolicy, the hook polls for the certificate - # for (timeoutSeconds - 30) to leave margin for ConfigMap creation. Increase - # this if cert-manager takes longer than 90 seconds to issue certificates. + # for (timeoutSeconds - 30) seconds, reserving 30 seconds for ConfigMap + # creation and cleanup. For example, timeoutSeconds=180 allows 150 seconds + # of polling. Increase this if cert-manager takes longer than 90 seconds to + # issue certificates. timeoutSeconds: 120 # -- Fail the helm install/upgrade if cert-manager does not issue the # certificate within the polling timeout. When true (default), the install diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 2e443e0b4a..258c7ba199 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -170,7 +170,9 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The chart will fail the install if you try to enable both `grpcRoute.backendTLSPolicy.enabled=true` and `server.tls.enableMtls=true` simultaneously. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 90 seconds to issue certificates, increase the timeout with `--set pkiInitJob.timeoutSeconds=180` (the hook will poll for 150 seconds with this value). By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. + +If cert-manager takes longer than 90 seconds to issue certificates, increase the job timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for `(timeoutSeconds - 30)` seconds, reserving 30 seconds for ConfigMap creation and cleanup. For example, `timeoutSeconds=180` allows 150 seconds of polling. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. ### Troubleshooting From 72e3aadbe946521dbab63d3a844c4d414793c3e8 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 16:15:29 -0400 Subject: [PATCH 16/27] docs(openshift): remove outdated two-stage install instructions Update OpenShift documentation to reflect that single-stage installs now work with cert-manager and BackendTLSPolicy. The Certificate resources run as pre-install hooks (weight -30) before certgen (weight -20), allowing the hook to poll for and find the issued certificates. Removed the outdated two-step process: 1. helm install (cert-manager issues cert, hook logs warning) 2. helm upgrade (hook creates ConfigMap) Replaced with current single-stage behavior: - Certificate resources created as pre-install hooks - certgen hook polls for up to 90 seconds (configurable) - Single helm install succeeds in most cases - Fails fast by default if timeout reached This brings openshift.mdx in line with the already-updated ingress.mdx documentation. Signed-off-by: Brandon Squizzato --- docs/kubernetes/openshift.mdx | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index d38347c8a7..d14aa5a86d 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -185,12 +185,9 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)). -**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. With cert-manager, the backend CA ConfigMap requires a two-step install because cert-manager creates the server certificate *after* the chart is installed: +**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are created as pre-install hooks (weight -30), then the certgen hook runs (weight -20) and polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases. -1. `helm install` — cert-manager issues the server certificate, but the certgen hook can't create the backend CA ConfigMap yet (logs a warning) -2. `helm upgrade` — certgen hook now reads the CA from the cert-manager-issued server certificate and creates the backend CA ConfigMap - -The gateway won't accept backend connections until step 2 completes. Wait for the cert-manager Certificate to be Ready before running upgrade. +If cert-manager takes longer than 90 seconds to issue certificates, increase the job timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for `(timeoutSeconds - 30)` seconds, reserving 30 seconds for ConfigMap creation and cleanup. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. #### Register over HTTPS From a1f57c416bc825e38ede004a49aa79d28f7e68f7 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 16:19:32 -0400 Subject: [PATCH 17/27] fix(helm): make pkiInitJob.timeoutSeconds the actual polling duration The timeout value now represents the actual polling time that users experience when waiting for cert-manager to issue certificates. The Job activeDeadlineSeconds is set to (timeoutSeconds + 30) to allow buffer time for ConfigMap creation and cleanup. Previously, the hook polled for (timeoutSeconds - 30) seconds, which was confusing when users set timeoutSeconds=180 and only got 150 seconds of actual polling. Updated documentation in values.yaml, ingress.mdx, and openshift.mdx to reflect the clearer behavior. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/templates/certgen.yaml | 4 ++-- deploy/helm/openshell/values.yaml | 13 +++++++------ docs/kubernetes/ingress.mdx | 4 ++-- docs/kubernetes/openshift.mdx | 4 ++-- 4 files changed, 13 insertions(+), 12 deletions(-) diff --git a/deploy/helm/openshell/templates/certgen.yaml b/deploy/helm/openshell/templates/certgen.yaml index 138c4d9cd2..392f977414 100644 --- a/deploy/helm/openshell/templates/certgen.yaml +++ b/deploy/helm/openshell/templates/certgen.yaml @@ -70,7 +70,7 @@ metadata: helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded spec: backoffLimit: 3 - activeDeadlineSeconds: {{ .Values.pkiInitJob.timeoutSeconds }} + activeDeadlineSeconds: {{ add .Values.pkiInitJob.timeoutSeconds 30 }} ttlSecondsAfterFinished: 300 template: metadata: @@ -123,7 +123,7 @@ spec: - --backend-ca-configmap-name={{ include "openshell.backendCaConfigMapName" . }} {{- if .Values.certManager.enabled }} - --backend-ca-source-secret={{ .Values.server.tls.certSecretName }} - - --backend-ca-poll-timeout-seconds={{ sub .Values.pkiInitJob.timeoutSeconds 30 }} + - --backend-ca-poll-timeout-seconds={{ .Values.pkiInitJob.timeoutSeconds }} {{- if .Values.pkiInitJob.failOnTimeout }} - --backend-ca-fail-on-timeout {{- end }} diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 074d099c34..64776c8482 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -468,12 +468,13 @@ pkiInitJob: serverDnsNames: [] # -- Extra IP SANs to append to the server certificate. serverIpAddresses: [] - # -- Maximum time in seconds for the certgen hook Job to run. When using - # cert-manager with BackendTLSPolicy, the hook polls for the certificate - # for (timeoutSeconds - 30) seconds, reserving 30 seconds for ConfigMap - # creation and cleanup. For example, timeoutSeconds=180 allows 150 seconds - # of polling. Increase this if cert-manager takes longer than 90 seconds to - # issue certificates. + # -- Maximum time in seconds for the certgen hook to poll for cert-manager + # certificates. When using cert-manager with BackendTLSPolicy, the hook + # polls for this many seconds waiting for the certificate to be issued, + # then creates the backend CA ConfigMap. The Job deadline is set to + # (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup. + # Increase this if cert-manager takes longer than 120 seconds to issue + # certificates. timeoutSeconds: 120 # -- Fail the helm install/upgrade if cert-manager does not issue the # certificate within the polling timeout. When true (default), the install diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 258c7ba199..dd7dca15ce 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -170,9 +170,9 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The chart will fail the install if you try to enable both `grpcRoute.backendTLSPolicy.enabled=true` and `server.tls.enableMtls=true` simultaneously. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. -If cert-manager takes longer than 90 seconds to issue certificates, increase the job timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for `(timeoutSeconds - 30)` seconds, reserving 30 seconds for ConfigMap creation and cleanup. For example, `timeoutSeconds=180` allows 150 seconds of polling. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. +If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. ### Troubleshooting diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index d14aa5a86d..1fee34fcc1 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -185,9 +185,9 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)). -**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are created as pre-install hooks (weight -30), then the certgen hook runs (weight -20) and polls for up to 90 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases. +**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are created as pre-install hooks (weight -30), then the certgen hook runs (weight -20) and polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases. -If cert-manager takes longer than 90 seconds to issue certificates, increase the job timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for `(timeoutSeconds - 30)` seconds, reserving 30 seconds for ConfigMap creation and cleanup. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. +If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. #### Register over HTTPS From fdec6edb2e759f2160d10709c71c6237627952ae Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 16:22:05 -0400 Subject: [PATCH 18/27] docs(helm): update values.yaml and README with correct polling duration Updated the caCertificateConfigMapName description to reflect that the hook polls for exactly pkiInitJob.timeoutSeconds seconds, not (timeoutSeconds - 30) seconds. Regenerated README.md with helm-docs. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/README.md | 4 +++- deploy/helm/openshell/values.yaml | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 31bb4ccdc8..23c953adcb 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -186,7 +186,7 @@ discovery endpoint or its TLS CA. | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | fullnameOverride | string | `""` | Override the full generated resource name. | -| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook waits for cert-manager to issue the server certificate first, so on first install the ConfigMap is created during the subsequent `helm upgrade` (run upgrade after cert-manager reconciles). | +| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. | | grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. | | grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (..svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | @@ -210,8 +210,10 @@ discovery endpoint or its TLS CA. | openshiftRoute.enabled | bool | `false` | Create an OpenShift Route with TLS passthrough. | | openshiftRoute.host | string | `""` | Hostname for the Route. Must match a SAN on the gateway's server cert. | | pkiInitJob.enabled | bool | `true` | Run a pre-install/pre-upgrade Job that creates gateway and client mTLS Secrets. When certManager.enabled=true, cert-manager owns TLS and this same hook runs in JWT-only mode even if pkiInitJob.enabled remains true. | +| pkiInitJob.failOnTimeout | bool | `true` | Fail the helm install/upgrade if cert-manager does not issue the certificate within the polling timeout. When true (default), the install fails immediately if the timeout is reached, providing clear feedback that BackendTLSPolicy is non-functional. When false, the hook succeeds with a warning and you can run `helm upgrade` after cert-manager issues the certificate to create the backend CA ConfigMap. If you set this to false and see "TLS error: Secret is not supplied by SDS" when connecting to the gateway, check if the TLS secret exists and run `helm upgrade` to create the ConfigMap. | | pkiInitJob.serverDnsNames | list | `[]` | Extra DNS SANs to append to the server certificate. | | pkiInitJob.serverIpAddresses | list | `[]` | Extra IP SANs to append to the server certificate. | +| pkiInitJob.timeoutSeconds | int | `120` | Maximum time in seconds for the certgen hook to poll for cert-manager certificates. When using cert-manager with BackendTLSPolicy, the hook polls for this many seconds waiting for the certificate to be issued, then creates the backend CA ConfigMap. The Job deadline is set to (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup. Increase this if cert-manager takes longer than 120 seconds to issue certificates. | | podAnnotations | object | `{}` | Extra annotations to add to the gateway pod. | | podLabels | object | `{}` | Extra labels to add to the gateway pod. | | podLifecycle.terminationGracePeriodSeconds | int | `5` | Grace period, in seconds, before Kubernetes terminates the gateway pod. | diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 64776c8482..a8cc11ba27 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -588,7 +588,7 @@ grpcRoute: # to validate the gateway pod's TLS certificate. Defaults to # -backend-ca when empty. The certgen hook auto-creates this: # with pkiInitJob (default), immediately on install/upgrade; with - # cert-manager, the hook polls for (pkiInitJob.timeoutSeconds - 30) seconds + # cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds # waiting for cert-manager to issue the server certificate, then creates the # ConfigMap. A single install usually succeeds; if cert-manager takes longer, # increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), From 75f6e1386a0afe241bc616f8fe9efb08b36b9efa Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 16:31:24 -0400 Subject: [PATCH 19/27] docs(kubernetes): add OIDC configuration to helm install and CLI examples Updated all helm install and openshell gateway add examples in ingress.mdx and openshift.mdx to include OIDC issuer and audience configuration. Examples now use concrete placeholder values: - OIDC issuer: https://keycloak.example.com/realms/openshell - OIDC audience: openshell-cli - Hostname: gateway.example.com - ClusterIssuer: letsencrypt-prod This makes it clearer how to configure OIDC authentication, which is required when using BackendTLSPolicy or HTTPS termination since the Gateway proxy cannot present client certificates to the backend. Signed-off-by: Brandon Squizzato --- docs/kubernetes/ingress.mdx | 14 +++++++++----- docs/kubernetes/openshift.mdx | 24 ++++++++++++++---------- 2 files changed, 23 insertions(+), 15 deletions(-) diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index dd7dca15ce..0149f817d7 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -120,8 +120,9 @@ helm upgrade --install openshell \ --set grpcRoute.gateway.listener.port=443 \ --set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \ --set server.disableTls=true \ - --set server.oidc.issuer=https:// \ - --set 'grpcRoute.hostnames[0]=' + --set server.oidc.issuer=https://keycloak.example.com/realms/openshell \ + --set server.oidc.audience=openshell-cli \ + --set 'grpcRoute.hostnames[0]=gateway.example.com' ``` Keep the certificate Secret in the release namespace. Referencing a Secret in another namespace requires a `ReferenceGrant`. @@ -129,7 +130,9 @@ Keep the certificate Secret in the release namespace. Referencing a Secret in an ### Register over HTTPS ```shell -openshell gateway add https:// --name production --oidc-issuer https:// +openshell gateway add https://gateway.example.com \ + --name production \ + --oidc-issuer https://keycloak.example.com/realms/openshell openshell status ``` @@ -164,8 +167,9 @@ helm upgrade --install openshell \ --set grpcRoute.gateway.listener.port=443 \ --set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \ --set grpcRoute.backendTLSPolicy.enabled=true \ - --set server.oidc.issuer=https:// \ - --set 'grpcRoute.hostnames[0]=' + --set server.oidc.issuer=https://keycloak.example.com/realms/openshell \ + --set server.oidc.audience=openshell-cli \ + --set 'grpcRoute.hostnames[0]=gateway.example.com' ``` Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The chart will fail the install if you try to enable both `grpcRoute.backendTLSPolicy.enabled=true` and `server.tls.enableMtls=true` simultaneously. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 1fee34fcc1..daa9306417 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -170,8 +170,10 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ --set grpcRoute.enabled=true \ --set grpcRoute.gateway.name=openshell-gateway \ --set grpcRoute.gateway.namespace=openshift-ingress \ - --set 'grpcRoute.hostnames[0]=' \ - --set grpcRoute.backendTLSPolicy.enabled=true + --set 'grpcRoute.hostnames[0]=gateway.example.com' \ + --set grpcRoute.backendTLSPolicy.enabled=true \ + --set server.oidc.issuer=https://keycloak.example.com/realms/openshell \ + --set server.oidc.audience=openshell-cli ``` | Override | Reason | @@ -192,7 +194,9 @@ If cert-manager takes longer than 120 seconds to issue certificates, increase th #### Register over HTTPS ```shell -openshell gateway add https:// --name openshift +openshell gateway add https://gateway.example.com \ + --name openshift \ + --oidc-issuer https://keycloak.example.com/realms/openshell openshell status ``` @@ -215,13 +219,13 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ --set securityContext.runAsUser=null \ --set server.disableTls=false \ --set certManager.enabled=true \ - --set certManager.serverIssuerRef.name= \ + --set certManager.serverIssuerRef.name=letsencrypt-prod \ --set certManager.serverIssuerRef.kind=ClusterIssuer \ - --set certManager.serverDnsNames[0]= \ + --set certManager.serverDnsNames[0]=gateway.example.com \ --set openshiftRoute.enabled=true \ - --set openshiftRoute.host= \ - --set server.oidc.issuer= \ - --set server.oidc.audience= + --set openshiftRoute.host=gateway.example.com \ + --set server.oidc.issuer=https://keycloak.example.com/realms/openshell \ + --set server.oidc.audience=openshell-cli ``` | Override | Reason | @@ -234,9 +238,9 @@ Register the gateway with the CLI over OIDC. Remote gateways authenticate CLI users via OIDC, not mTLS — see [Access Control](/kubernetes/access-control): ```shell -openshell gateway add https:// \ +openshell gateway add https://gateway.example.com \ --name openshift \ - --oidc-issuer + --oidc-issuer https://keycloak.example.com/realms/openshell openshell gateway login openshift ``` From 200b23a39e7b55657645d83d268f4fbdf2e3a194 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 21 Aug 2026 16:34:29 -0400 Subject: [PATCH 20/27] docs(kubernetes): explicitly list OIDC client ID in gateway add examples Added --oidc-client-id openshell-cli to all openshell gateway add commands in ingress.mdx and openshift.mdx, making the default client ID explicit in the examples even though it's the CLI default. This improves clarity and helps users understand the complete OIDC configuration needed for gateway registration. Signed-off-by: Brandon Squizzato --- docs/kubernetes/ingress.mdx | 3 ++- docs/kubernetes/openshift.mdx | 6 ++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 0149f817d7..afe84a95d8 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -132,7 +132,8 @@ Keep the certificate Secret in the release namespace. Referencing a Secret in an ```shell openshell gateway add https://gateway.example.com \ --name production \ - --oidc-issuer https://keycloak.example.com/realms/openshell + --oidc-issuer https://keycloak.example.com/realms/openshell \ + --oidc-client-id openshell-cli openshell status ``` diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index daa9306417..8bc8eaaaea 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -196,7 +196,8 @@ If cert-manager takes longer than 120 seconds to issue certificates, increase th ```shell openshell gateway add https://gateway.example.com \ --name openshift \ - --oidc-issuer https://keycloak.example.com/realms/openshell + --oidc-issuer https://keycloak.example.com/realms/openshell \ + --oidc-client-id openshell-cli openshell status ``` @@ -240,7 +241,8 @@ users via OIDC, not mTLS — see [Access Control](/kubernetes/access-control): ```shell openshell gateway add https://gateway.example.com \ --name openshift \ - --oidc-issuer https://keycloak.example.com/realms/openshell + --oidc-issuer https://keycloak.example.com/realms/openshell \ + --oidc-client-id openshell-cli openshell gateway login openshift ``` From cccb0498d43cdb6836c4c606651dca4f355bcafd Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Wed, 26 Aug 2026 17:29:37 -0400 Subject: [PATCH 21/27] fix(helm): address PR review feedback for BackendTLSPolicy - Read backend CA from the authoritative server Secret instead of the in-memory PKI bundle so enabling BackendTLSPolicy on an existing release uses the CA that actually signed the server certificate. - Reconcile the backend CA ConfigMap on every hook run (compare and update) instead of skipping when it already exists, so CA rotations propagate automatically. - Remove hook annotations from cert-manager Issuer/Certificate resources so they remain regular release objects managed by Helm lifecycle. Split the cert-manager backend CA ConfigMap creation into a separate post-install/post-upgrade hook Job that polls after cert-manager Certificate resources are applied. - Update architecture/gateway.md, docs/reference/gateway-config.mdx, debug-openshell-cluster skill, and helm-dev-environment skill with BackendTLSPolicy, backend CA ConfigMap, enableMtls, and timeout documentation. Signed-off-by: Brandon Squizzato Signed-off-by: Brandon Squizzato --- .agents/skills/helm-dev-environment/SKILL.md | 29 +++++ architecture/gateway.md | 19 +++ crates/openshell-server/src/certgen.rs | 112 +++++++++++------- .../openshell/templates/cert-manager-pki.yaml | 18 --- deploy/helm/openshell/templates/certgen.yaml | 61 +++++++++- docs/reference/gateway-config.mdx | 3 + skills/debug-openshell-cluster/SKILL.md | 25 ++++ 7 files changed, 203 insertions(+), 64 deletions(-) diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index 780cf8b9c6..15198a891f 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -225,6 +225,35 @@ Envoy Gateway is already installed by Skaffold (the `envoy-gateway` Helm release service for the proxy; klipper-lb binds it to hostPort 80, reachable via the `8080:80` load balancer port mapping. +### BackendTLSPolicy (end-to-end TLS) + +To enable end-to-end TLS between the Gateway proxy and the gateway pod, add +BackendTLSPolicy values to the Helm install: + +```bash +helm upgrade --install openshell deploy/helm/openshell \ + --set grpcRoute.enabled=true \ + --set grpcRoute.backendTLSPolicy.enabled=true \ + --set server.tls.enableMtls=false \ + ... +``` + +This requires `server.tls.enableMtls=false` because ingress proxies cannot +present client certificates to the backend. The certgen hook creates a backend +CA ConfigMap from the server Secret's `ca.crt` key. With cert-manager, a +separate post-install Job polls for the cert-manager-issued certificate (up to +`pkiInitJob.timeoutSeconds`); with built-in PKI the ConfigMap is created in the +same pre-install hook. The ConfigMap is reconciled on every upgrade so CA +rotations propagate automatically. + +Key Helm values: +- `grpcRoute.backendTLSPolicy.enabled`: create the BackendTLSPolicy resource +- `grpcRoute.backendTLSPolicy.caCertificateConfigMapName`: override ConfigMap name +- `grpcRoute.backendTLSPolicy.hostname`: override backend validation hostname +- `server.tls.enableMtls`: must be `false` for BackendTLSPolicy +- `pkiInitJob.timeoutSeconds`: polling duration for cert-manager mode +- `pkiInitJob.failOnTimeout`: fail install if cert-manager times out + ### Keycloak OIDC One-time setup — only needed once per cluster lifetime: diff --git a/architecture/gateway.md b/architecture/gateway.md index 8fc28dfb21..b1f6c6341b 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -757,6 +757,25 @@ requested present -> generate and write. This guards continuity across restarts and upgrades while still recovering cleanly if an operator deletes everything and starts over. +When `grpcRoute.backendTLSPolicy.enabled=true`, the certgen hook also creates a +`ConfigMap` containing the CA certificate (`ca.crt`) used by the Gateway proxy +to validate the backend pod's TLS certificate. The CA is always read from the +authoritative server Secret (not the in-memory bundle) so that enabling +BackendTLSPolicy on an existing release uses the CA that actually signed the +server certificate. The ConfigMap is reconciled on every hook run: if the CA +changes (rotation, re-issue), the ConfigMap is updated in place. In built-in PKI +mode the ConfigMap is created in the same pre-install hook. In cert-manager +mode, a separate post-install/post-upgrade hook Job polls for the cert-manager- +issued server Secret and then creates or updates the ConfigMap, because +cert-manager Certificate resources are regular release objects applied after +pre-install hooks. + +The `server.tls.enableMtls` value controls whether the gateway requires client +certificates. When `enableMtls` is `false`, the gateway runs HTTPS-only without +client certificate verification (use OIDC for identity instead). BackendTLSPolicy +requires `enableMtls=false` because the ingress proxy cannot present client +certificates to the backend. + Operators who manage TLS PKI with cert-manager enable `certManager.enabled`; cert-manager takes precedence over built-in TLS generation and the chart still renders the JWT-only hook. Operators who pre-create all TLS and JWT Secrets can diff --git a/crates/openshell-server/src/certgen.rs b/crates/openshell-server/src/certgen.rs index b42de5c422..764ceba98c 100644 --- a/crates/openshell-server/src/certgen.rs +++ b/crates/openshell-server/src/certgen.rs @@ -80,9 +80,9 @@ pub struct CertgenArgs { dry_run: bool, /// Name of a `ConfigMap` to create containing the CA certificate (key: ca.crt) - /// for `BackendTLSPolicy` backend validation. In full PKI mode, the CA comes - /// from the generated bundle. In --jwt-only mode, the CA is read from - /// --backend-ca-source-secret. + /// for `BackendTLSPolicy` backend validation. The CA is always read from the + /// authoritative server Secret: --server-secret-name in full PKI mode, + /// --backend-ca-source-secret in --jwt-only mode. #[arg(long, value_name = "NAME")] backend_ca_configmap_name: Option, @@ -127,7 +127,7 @@ pub async fn run(args: CertgenArgs) -> Result<()> { run_kubernetes(&args, &bundle).await?; if let Some(ref cm_name) = args.backend_ca_configmap_name { - create_backend_ca_configmap_if_needed(&args, &bundle, cm_name).await?; + create_backend_ca_configmap_if_needed(&args, cm_name).await?; } Ok(()) @@ -326,9 +326,21 @@ async fn create_tls_secrets( Ok(()) } +fn extract_ca_from_secret(secret: &Secret, name: &str) -> Result { + let data = secret + .data + .as_ref() + .ok_or_else(|| miette::miette!("secret {name} has no data"))?; + let ca = data + .get("ca.crt") + .ok_or_else(|| miette::miette!("secret {name} has no ca.crt key"))?; + String::from_utf8(ca.0.clone()) + .into_diagnostic() + .wrap_err("ca.crt is not valid UTF-8") +} + async fn create_backend_ca_configmap_if_needed( args: &CertgenArgs, - bundle: &PkiBundle, configmap_name: &str, ) -> Result<()> { let namespace = args @@ -340,31 +352,13 @@ async fn create_backend_ca_configmap_if_needed( .await .into_diagnostic() .wrap_err("failed to construct Kubernetes client for backend CA ConfigMap")?; - let api: Api = Api::namespaced(client.clone(), namespace); - - if api - .get_opt(configmap_name) - .await - .into_diagnostic() - .wrap_err_with(|| format!("failed to read configmap {configmap_name}"))? - .is_some() - { - info!( - namespace = %namespace, - configmap = %configmap_name, - "Backend CA ConfigMap already exists, skipping." - ); - return Ok(()); - } - - let ca_pem = if !args.jwt_only { - bundle.ca_cert_pem.clone() - } else if let Some(source_secret) = &args.backend_ca_source_secret { - let secret_api: Api = Api::namespaced(client, namespace); + let secret_api: Api = Api::namespaced(client.clone(), namespace); + let cm_api: Api = Api::namespaced(client, namespace); - // Poll for the cert-manager Secret with timeout to handle cert issuance delay. - // The Helm chart sets this to (Job activeDeadlineSeconds - 30) leaving - // margin for ConfigMap creation and hook completion. + // Resolve the CA from the authoritative server Secret rather than the + // in-memory bundle so upgrades that enable BackendTLSPolicy on an + // existing release use the CA that actually signed the server cert. + let ca_pem = if let Some(source_secret) = &args.backend_ca_source_secret { let poll_timeout = std::time::Duration::from_secs(args.backend_ca_poll_timeout_seconds); let poll_interval = std::time::Duration::from_secs(2); let start = std::time::Instant::now(); @@ -416,23 +410,58 @@ async fn create_backend_ca_configmap_if_needed( elapsed_secs = start.elapsed().as_secs(), "cert-manager TLS certificate found." ); - - let data = secret - .data - .ok_or_else(|| miette::miette!("secret {source_secret} has no data"))?; - let ca = data - .get("ca.crt") - .ok_or_else(|| miette::miette!("secret {source_secret} has no ca.crt key"))?; - String::from_utf8(ca.0.clone()) + extract_ca_from_secret(&secret, source_secret)? + } else if let Some(server_secret) = &args.server_secret_name { + let secret = secret_api + .get(server_secret) + .await .into_diagnostic() - .wrap_err("ca.crt is not valid UTF-8")? + .wrap_err_with(|| format!("failed to read server secret {server_secret}"))?; + extract_ca_from_secret(&secret, server_secret)? } else { return Err(miette::miette!( - "--backend-ca-source-secret is required with --jwt-only \ - and --backend-ca-configmap-name" + "--backend-ca-source-secret or --server-secret-name is required \ + with --backend-ca-configmap-name" )); }; + // Reconcile: create or update so the backend CA stays current across + // CA rotations and upgrades. + if let Some(existing) = cm_api + .get_opt(configmap_name) + .await + .into_diagnostic() + .wrap_err_with(|| format!("failed to read configmap {configmap_name}"))? + { + let up_to_date = existing + .data + .as_ref() + .and_then(|d| d.get("ca.crt")) + .map(String::as_str) + == Some(&ca_pem); + if up_to_date { + info!( + namespace = %namespace, + configmap = %configmap_name, + "Backend CA ConfigMap is up-to-date, skipping." + ); + return Ok(()); + } + let mut updated = existing; + updated.data = Some(BTreeMap::from([("ca.crt".to_string(), ca_pem)])); + cm_api + .replace(configmap_name, &PostParams::default(), &updated) + .await + .into_diagnostic() + .wrap_err_with(|| format!("failed to update configmap {configmap_name}"))?; + info!( + namespace = %namespace, + configmap = %configmap_name, + "Backend CA ConfigMap updated with current CA." + ); + return Ok(()); + } + let configmap = ConfigMap { metadata: ObjectMeta { name: Some(configmap_name.to_string()), @@ -442,7 +471,8 @@ async fn create_backend_ca_configmap_if_needed( ..Default::default() }; - api.create(&PostParams::default(), &configmap) + cm_api + .create(&PostParams::default(), &configmap) .await .into_diagnostic() .wrap_err_with(|| format!("failed to create configmap {configmap_name}"))?; diff --git a/deploy/helm/openshell/templates/cert-manager-pki.yaml b/deploy/helm/openshell/templates/cert-manager-pki.yaml index 54b4bfef5d..838534e62e 100644 --- a/deploy/helm/openshell/templates/cert-manager-pki.yaml +++ b/deploy/helm/openshell/templates/cert-manager-pki.yaml @@ -8,9 +8,6 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} - annotations: - helm.sh/hook: pre-install,pre-upgrade - helm.sh/hook-weight: "-30" spec: selfSigned: {} --- @@ -21,9 +18,6 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} - annotations: - helm.sh/hook: pre-install,pre-upgrade - helm.sh/hook-weight: "-30" spec: isCA: true commonName: openshell-ca @@ -43,9 +37,6 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} - annotations: - helm.sh/hook: pre-install,pre-upgrade - helm.sh/hook-weight: "-30" spec: ca: secretName: {{ .Values.certManager.caSecretName | quote }} @@ -75,9 +66,6 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} - annotations: - helm.sh/hook: pre-install,pre-upgrade - helm.sh/hook-weight: "-30" spec: secretName: {{ .Values.server.tls.certSecretName | quote }} duration: {{ .Values.certManager.certificateDuration | quote }} @@ -119,9 +107,6 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} - annotations: - helm.sh/hook: pre-install,pre-upgrade - helm.sh/hook-weight: "-30" spec: secretName: {{ include "openshell.fullname" . }}-server-external-tls duration: {{ .Values.certManager.certificateDuration | quote }} @@ -153,9 +138,6 @@ metadata: namespace: {{ .Release.Namespace }} labels: {{- include "openshell.labels" . | nindent 4 }} - annotations: - helm.sh/hook: pre-install,pre-upgrade - helm.sh/hook-weight: "-30" spec: secretName: {{ .Values.server.tls.clientTlsSecretName | quote }} duration: {{ .Values.certManager.certificateDuration | quote }} diff --git a/deploy/helm/openshell/templates/certgen.yaml b/deploy/helm/openshell/templates/certgen.yaml index 392f977414..b2ce8d2a82 100644 --- a/deploy/helm/openshell/templates/certgen.yaml +++ b/deploy/helm/openshell/templates/certgen.yaml @@ -34,7 +34,7 @@ rules: {{- if .Values.grpcRoute.backendTLSPolicy.enabled }} - apiGroups: [""] resources: ["configmaps"] - verbs: ["get", "create"] + verbs: ["get", "create", "update"] {{- end }} --- apiVersion: rbac.authorization.k8s.io/v1 @@ -119,14 +119,65 @@ spec: - --server-san={{ . }} {{- end }} {{- end }} - {{- if .Values.grpcRoute.backendTLSPolicy.enabled }} + {{- if and .Values.grpcRoute.backendTLSPolicy.enabled (not .Values.certManager.enabled) }} + - --backend-ca-configmap-name={{ include "openshell.backendCaConfigMapName" . }} + {{- end }} +{{- if and .Values.certManager.enabled .Values.grpcRoute.backendTLSPolicy.enabled }} +--- +# Post-install Job that polls for the cert-manager-issued server Secret and +# creates / reconciles the backend CA ConfigMap. Runs after regular resources +# (including the cert-manager Certificate objects) are applied so cert-manager +# has started issuing before the poll begins. +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ $hookName }}-backend-ca + namespace: {{ $ns }} + labels: + {{- include "openshell.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + backoffLimit: 3 + activeDeadlineSeconds: {{ add .Values.pkiInitJob.timeoutSeconds 30 }} + ttlSecondsAfterFinished: 300 + template: + metadata: + labels: + {{- include "openshell.selectorLabels" . | nindent 8 }} + spec: + restartPolicy: OnFailure + serviceAccountName: {{ $hookName }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: certgen + image: {{ include "openshell.image" . | quote }} + imagePullPolicy: {{ .Values.image.pullPolicy }} + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + env: + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + command: ["/usr/local/bin/openshell-gateway"] + args: + - generate-certs + - --jwt-only + - --jwt-secret-name={{ include "openshell.sandboxJwtSecretName" . }} - --backend-ca-configmap-name={{ include "openshell.backendCaConfigMapName" . }} - {{- if .Values.certManager.enabled }} - --backend-ca-source-secret={{ .Values.server.tls.certSecretName }} - --backend-ca-poll-timeout-seconds={{ .Values.pkiInitJob.timeoutSeconds }} {{- if .Values.pkiInitJob.failOnTimeout }} - --backend-ca-fail-on-timeout {{- end }} - {{- end }} - {{- end }} +{{- end }} {{- end }} diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index e2d48087cd..f9a3f00321 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -522,6 +522,9 @@ compute_driver = "kubernetes" [openshell.gateway.tls] cert_path = "/etc/openshell-tls/server/tls.crt" key_path = "/etc/openshell-tls/server/tls.key" +# client_ca_path is only rendered when server.tls.enableMtls is true (the +# default). When enableMtls is false — required for BackendTLSPolicy — the +# gateway runs HTTPS-only and this line is omitted by Helm. client_ca_path = "/etc/openshell-tls/client-ca/ca.crt" # When cert-manager serverIssuerRef is configured, these are populated by Helm: # external_cert_path = "/etc/openshell-tls/server-external/tls.crt" diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 360bf28ee4..577b801a83 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -446,6 +446,31 @@ label, supervisor env vars `OPENSHELL_K8S_SA_TOKEN_FILE` and `OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET`, plus both the projected `openshell-sa-token` volume and the `spiffe-workload-api` CSI volume. +If `grpcRoute.backendTLSPolicy.enabled=true`, the Gateway proxy validates the +backend pod's TLS certificate against a CA in a ConfigMap. Check that the +ConfigMap exists and contains the correct CA, that `enableMtls` is disabled, +and that the BackendTLSPolicy resource is present: + +```bash +kubectl -n openshell get backendtlspolicy +kubectl -n openshell get configmap openshell-backend-ca -o yaml +helm -n openshell get values openshell | grep -E 'backendTLSPolicy|enableMtls|failOnTimeout|timeoutSeconds|caCertificateConfigMapName' +``` + +If the ConfigMap is missing after a cert-manager install, the post-install +certgen hook may have timed out waiting for cert-manager to issue the server +certificate. Check the certgen Job logs: + +```bash +kubectl -n openshell get jobs | grep certgen +kubectl -n openshell logs job/openshell-certgen-backend-ca +``` + +Increase `pkiInitJob.timeoutSeconds` and run `helm upgrade` to retry. If the +Gateway proxy reports `TLS error: Secret is not supplied by SDS` or similar +backend TLS errors, the ConfigMap CA likely does not match the server +certificate CA — verify both are from the same issuer. + Check the image references currently used by the gateway deployment: ```bash From 605247bec4153b6df1bc03da8eb510a9f3c88bb1 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 4 Sep 2026 13:00:22 -0400 Subject: [PATCH 22/27] fix(docs): resolve markdown lint errors in helm README and kubernetes docs Escape inline HTML angle brackets in README.md template placeholders, remove trailing spaces, and add blank lines around fenced code blocks in numbered lists. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/README.md | 4 ++-- docs/kubernetes/ingress.mdx | 5 ++++- docs/kubernetes/openshift.mdx | 2 +- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 23c953adcb..91181e4c2f 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -186,9 +186,9 @@ discovery endpoint or its TLS CA. | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | fullnameOverride | string | `""` | Override the full generated resource name. | -| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. | +| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to \-backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. | | grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. | -| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (..svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | +| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (\.\.svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | | grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. | diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index afe84a95d8..0623c6abd8 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -175,7 +175,7 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The chart will fail the install if you try to enable both `grpcRoute.backendTLSPolicy.enabled=true` and `server.tls.enableMtls=true` simultaneously. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. @@ -184,16 +184,19 @@ If cert-manager takes longer than 120 seconds to issue certificates, increase th If you see the error `remote connection failure, transport failure reason: TLS error: Secret is not supplied by SDS` when connecting through the Gateway: 1. Check if the backend CA ConfigMap exists: + ```shell kubectl get configmap -backend-ca -n ``` 2. If the ConfigMap is missing, verify the TLS secret exists: + ```shell kubectl get secret -server-tls -n ``` 3. If the secret exists but the ConfigMap doesn't, run `helm upgrade` to create it: + ```shell helm upgrade oci://ghcr.io/nvidia/openshell/helm-chart \ --reuse-values --namespace diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index 8bc8eaaaea..e3aa48d6ab 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -187,7 +187,7 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)). -**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are created as pre-install hooks (weight -30), then the certgen hook runs (weight -20) and polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases. +**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are created as pre-install hooks (weight -30), then the certgen hook runs (weight -20) and polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases. If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. From 46f8b2ede7e0a904e522f69e3b47e8c6ac27a527 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Wed, 9 Sep 2026 16:42:11 -0400 Subject: [PATCH 23/27] Update docs --- deploy/helm/openshell/README.md | 4 +-- mise.lock | 45 +++++++-------------------------- 2 files changed, 11 insertions(+), 38 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 91181e4c2f..23c953adcb 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -186,9 +186,9 @@ discovery endpoint or its TLS CA. | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | fullnameOverride | string | `""` | Override the full generated resource name. | -| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to \-backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. | +| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. | | grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. | -| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (\.\.svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | +| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (..svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | | grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. | diff --git a/mise.lock b/mise.lock index 522084b559..e7e30626e8 100644 --- a/mise.lock +++ b/mise.lock @@ -119,11 +119,6 @@ checksum = "sha256:f73a5c39f96bb6ebb89cc7915cf182260d4cbf30765322c5e793d0fe8bd80 url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-aarch64-unknown-linux-musl.tar.gz" url_api = "https://api.github.com/repos/mozilla/sccache/releases/assets/452060468" -[tools."github:mozilla/sccache"."platforms.linux-x64"] -checksum = "sha256:aec995a83ad3dff3d14b6314e08858b7b73d35ca85a5bcf3d3a9ec07dee35588" -url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-x86_64-unknown-linux-musl.tar.gz" -url_api = "https://api.github.com/repos/mozilla/sccache/releases/assets/452060682" - [tools."github:mozilla/sccache"."platforms.macos-arm64"] checksum = "sha256:ded590cae2c72042c61178632906bef62d635fa20d45f8b22110a2241f430960" url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-aarch64-apple-darwin.tar.gz" @@ -134,39 +129,17 @@ checksum = "sha256:b8514ed7552e148b0a032114f745118dcb801791adafafeaf9935e4bfb0ed url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-x86_64-pc-windows-msvc.zip" url_api = "https://api.github.com/repos/mozilla/sccache/releases/assets/452060720" -[[tools."github:nextest-rs/nextest"]] -version = "cargo-nextest-0.9.143" -backend = "github:nextest-rs/nextest" - -[tools."github:nextest-rs/nextest"."platforms.linux-arm64"] -checksum = "sha256:2a64b3566a92508550a7ab29c3e8db25472ca37730ecb4d22100b6aa440c2a68" -url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-aarch64-unknown-linux-gnu.tar.gz" -url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501881925" -provenance = "github-attestations" - -[tools."github:nextest-rs/nextest"."platforms.linux-x64"] -checksum = "sha256:66786b9abe23920d022a182d1416b1bbc8130dd4872a9553d76985a1708dcd1e" -url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-x86_64-unknown-linux-gnu.tar.gz" -url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501882467" -provenance = "github-attestations" - -[tools."github:nextest-rs/nextest"."platforms.macos-arm64"] -checksum = "sha256:4830d430411148d17602a75cc880bfb4dc8dac153dea59a48a2ef4cc93577f07" -url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-universal-apple-darwin.tar.gz" -url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501885637" -provenance = "github-attestations" +[[tools."github:mozilla/sccache"]] +version = "0.16.0" +backend = "github:mozilla/sccache" -[tools."github:nextest-rs/nextest"."platforms.windows-arm64"] -checksum = "sha256:58c1637ba2396e6c556aa0092f9aa4388695594b8ddda5a4b8b39212574678ce" -url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-aarch64-pc-windows-msvc.zip" -url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501886128" -provenance = "github-attestations" +[tools."github:mozilla/sccache".options] +asset_pattern = "sccache-v*x86_64*linux*.tar.gz" -[tools."github:nextest-rs/nextest"."platforms.windows-x64"] -checksum = "sha256:c670ba18e8731fd2eff33a47af33a0fa53d1afa6d0678344e82dc6f8fc7344ac" -url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-x86_64-pc-windows-msvc.zip" -url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501885754" -provenance = "github-attestations" +[tools."github:mozilla/sccache"."platforms.linux-x64"] +checksum = "sha256:aec995a83ad3dff3d14b6314e08858b7b73d35ca85a5bcf3d3a9ec07dee35588" +url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-x86_64-unknown-linux-musl.tar.gz" +url_api = "https://api.github.com/repos/mozilla/sccache/releases/assets/452060682" [[tools."github:rust-cross/cargo-zigbuild"]] version = "0.22.3" From 06eed16a6100da9a7c0f1ffb63d490262a555c0b Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 11 Sep 2026 11:42:07 -0400 Subject: [PATCH 24/27] fix(helm): escape inline HTML in values.yaml descriptions and sync mise lockfile Wrap `` and `` template placeholders in backticks so markdownlint does not flag them as inline HTML (MD033). Regenerate mise.lock to match current mise.toml after rebase onto main. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/README.md | 4 ++-- deploy/helm/openshell/values.yaml | 4 ++-- mise.lock | 40 ++++++++++++++++++++++++++----- 3 files changed, 38 insertions(+), 10 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 23c953adcb..659a0f5d0b 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -186,9 +186,9 @@ discovery endpoint or its TLS CA. | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | fullnameOverride | string | `""` | Override the full generated resource name. | -| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to -backend-ca when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. | +| grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to `-backend-ca` when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. | | grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. | -| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (..svc.cluster.local) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | +| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (`..svc.cluster.local`) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". | | grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. | diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index a8cc11ba27..b2a1a1a8b0 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -586,7 +586,7 @@ grpcRoute: enabled: false # -- Name of the ConfigMap containing the CA certificate (key: ca.crt) used # to validate the gateway pod's TLS certificate. Defaults to - # -backend-ca when empty. The certgen hook auto-creates this: + # `-backend-ca` when empty. The certgen hook auto-creates this: # with pkiInitJob (default), immediately on install/upgrade; with # cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds # waiting for cert-manager to issue the server certificate, then creates the @@ -597,7 +597,7 @@ grpcRoute: caCertificateConfigMapName: "" # -- Hostname the Gateway proxy validates against the backend's TLS # certificate SAN. Defaults to the service FQDN - # (..svc.cluster.local) when empty, which matches + # (`..svc.cluster.local`) when empty, which matches # the SAN included by both cert-manager and the pkiInitJob. hostname: "" diff --git a/mise.lock b/mise.lock index e7e30626e8..9391a41ff6 100644 --- a/mise.lock +++ b/mise.lock @@ -97,6 +97,7 @@ checksum = "sha256:0e91737aee2b5baf1d255b959630194a302335d848ff97bb07921eb6205b5 url = "https://github.com/anchore/syft/releases/download/v1.44.0/syft_1.44.0_linux_amd64.tar.gz" url_api = "https://api.github.com/repos/anchore/syft/releases/assets/410001183" provenance = "github-attestations" +provenance_verified = true [tools."github:anchore/syft"."platforms.macos-arm64"] checksum = "sha256:24e4d34078ae81da7c82539616f0ccac3e226cf4f74a38ce6fb3463619e50a55" @@ -141,6 +142,33 @@ checksum = "sha256:aec995a83ad3dff3d14b6314e08858b7b73d35ca85a5bcf3d3a9ec07dee35 url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-x86_64-unknown-linux-musl.tar.gz" url_api = "https://api.github.com/repos/mozilla/sccache/releases/assets/452060682" +[[tools."github:nextest-rs/nextest"]] +version = "cargo-nextest-0.9.143" +backend = "github:nextest-rs/nextest" + +[tools."github:nextest-rs/nextest"."platforms.linux-arm64"] +checksum = "sha256:2a64b3566a92508550a7ab29c3e8db25472ca37730ecb4d22100b6aa440c2a68" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-aarch64-unknown-linux-gnu.tar.gz" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501881925" +provenance = "github-attestations" + +[tools."github:nextest-rs/nextest"."platforms.linux-x64"] +checksum = "sha256:66786b9abe23920d022a182d1416b1bbc8130dd4872a9553d76985a1708dcd1e" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-x86_64-unknown-linux-gnu.tar.gz" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501882467" + +[tools."github:nextest-rs/nextest"."platforms.macos-arm64"] +checksum = "sha256:4830d430411148d17602a75cc880bfb4dc8dac153dea59a48a2ef4cc93577f07" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-universal-apple-darwin.tar.gz" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501885637" +provenance = "github-attestations" + +[tools."github:nextest-rs/nextest"."platforms.windows-x64"] +checksum = "sha256:c670ba18e8731fd2eff33a47af33a0fa53d1afa6d0678344e82dc6f8fc7344ac" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-x86_64-pc-windows-msvc.zip" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501885754" +provenance = "github-attestations" + [[tools."github:rust-cross/cargo-zigbuild"]] version = "0.22.3" backend = "github:rust-cross/cargo-zigbuild" @@ -422,15 +450,15 @@ version = "0.10.12" backend = "aqua:astral-sh/uv" [tools.uv."platforms.linux-arm64"] -checksum = "sha256:55bd1c1c10ec8b95a8c184f5e18b566703c6ab105f0fc118aaa4d748aabf28e4" -url = "https://github.com/astral-sh/uv/releases/download/0.10.12/uv-aarch64-unknown-linux-musl.tar.gz" -url_api = "https://api.github.com/repos/astral-sh/uv/releases/assets/377491942" +checksum = "sha256:0ed7d20f49f6b9b60d45fdfcac28f3ac01a671a6ef08672401ed2833423fea2a" +url = "https://github.com/astral-sh/uv/releases/download/0.10.12/uv-aarch64-unknown-linux-gnu.tar.gz" +url_api = "https://api.github.com/repos/astral-sh/uv/releases/assets/377491938" provenance = "github-attestations" [tools.uv."platforms.linux-x64"] -checksum = "sha256:adccf40b5d1939a5e0093081ec2307ea24235adf7c2d96b122c561fa37711c46" -url = "https://github.com/astral-sh/uv/releases/download/0.10.12/uv-x86_64-unknown-linux-musl.tar.gz" -url_api = "https://api.github.com/repos/astral-sh/uv/releases/assets/377491998" +checksum = "sha256:ec72570c9d1f33021aa80b176d7baba390de2cfeb1abcbefca346d563bf17484" +url = "https://github.com/astral-sh/uv/releases/download/0.10.12/uv-x86_64-unknown-linux-gnu.tar.gz" +url_api = "https://api.github.com/repos/astral-sh/uv/releases/assets/377491996" provenance = "github-attestations" [tools.uv."platforms.macos-arm64"] From 93c879d60faaa2934ce3c8efbcd841e254fb35f9 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 11 Sep 2026 14:35:22 -0400 Subject: [PATCH 25/27] Run 'mise lock' --- mise.lock | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mise.lock b/mise.lock index 9391a41ff6..5e81c99980 100644 --- a/mise.lock +++ b/mise.lock @@ -156,6 +156,8 @@ provenance = "github-attestations" checksum = "sha256:66786b9abe23920d022a182d1416b1bbc8130dd4872a9553d76985a1708dcd1e" url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-x86_64-unknown-linux-gnu.tar.gz" url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501882467" +provenance = "github-attestations" +provenance_verified = true [tools."github:nextest-rs/nextest"."platforms.macos-arm64"] checksum = "sha256:4830d430411148d17602a75cc880bfb4dc8dac153dea59a48a2ef4cc93577f07" From 4ae7135367d42c3f2441e3082fe205d8a85c1435 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 11 Sep 2026 15:06:10 -0400 Subject: [PATCH 26/27] fix: align mise.lock with CI mise version output The lockfile was regenerated locally with mise 2026.8.10 which resolves uv Linux artifacts to gnu variants and adds provenance_verified fields, but CI uses v2026.4.25 which produces musl variants without those fields. Signed-off-by: Brandon Squizzato --- mise.lock | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mise.lock b/mise.lock index 5e81c99980..a5e38be2c2 100644 --- a/mise.lock +++ b/mise.lock @@ -97,7 +97,6 @@ checksum = "sha256:0e91737aee2b5baf1d255b959630194a302335d848ff97bb07921eb6205b5 url = "https://github.com/anchore/syft/releases/download/v1.44.0/syft_1.44.0_linux_amd64.tar.gz" url_api = "https://api.github.com/repos/anchore/syft/releases/assets/410001183" provenance = "github-attestations" -provenance_verified = true [tools."github:anchore/syft"."platforms.macos-arm64"] checksum = "sha256:24e4d34078ae81da7c82539616f0ccac3e226cf4f74a38ce6fb3463619e50a55" @@ -120,6 +119,11 @@ checksum = "sha256:f73a5c39f96bb6ebb89cc7915cf182260d4cbf30765322c5e793d0fe8bd80 url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-aarch64-unknown-linux-musl.tar.gz" url_api = "https://api.github.com/repos/mozilla/sccache/releases/assets/452060468" +[tools."github:mozilla/sccache"."platforms.linux-x64"] +checksum = "sha256:aec995a83ad3dff3d14b6314e08858b7b73d35ca85a5bcf3d3a9ec07dee35588" +url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-x86_64-unknown-linux-musl.tar.gz" +url_api = "https://api.github.com/repos/mozilla/sccache/releases/assets/452060682" + [tools."github:mozilla/sccache"."platforms.macos-arm64"] checksum = "sha256:ded590cae2c72042c61178632906bef62d635fa20d45f8b22110a2241f430960" url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-aarch64-apple-darwin.tar.gz" @@ -157,7 +161,6 @@ checksum = "sha256:66786b9abe23920d022a182d1416b1bbc8130dd4872a9553d76985a1708dc url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-x86_64-unknown-linux-gnu.tar.gz" url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501882467" provenance = "github-attestations" -provenance_verified = true [tools."github:nextest-rs/nextest"."platforms.macos-arm64"] checksum = "sha256:4830d430411148d17602a75cc880bfb4dc8dac153dea59a48a2ef4cc93577f07" @@ -452,15 +455,13 @@ version = "0.10.12" backend = "aqua:astral-sh/uv" [tools.uv."platforms.linux-arm64"] -checksum = "sha256:0ed7d20f49f6b9b60d45fdfcac28f3ac01a671a6ef08672401ed2833423fea2a" -url = "https://github.com/astral-sh/uv/releases/download/0.10.12/uv-aarch64-unknown-linux-gnu.tar.gz" -url_api = "https://api.github.com/repos/astral-sh/uv/releases/assets/377491938" +checksum = "sha256:55bd1c1c10ec8b95a8c184f5e18b566703c6ab105f0fc118aaa4d748aabf28e4" +url = "https://github.com/astral-sh/uv/releases/download/0.10.12/uv-aarch64-unknown-linux-musl.tar.gz" provenance = "github-attestations" [tools.uv."platforms.linux-x64"] -checksum = "sha256:ec72570c9d1f33021aa80b176d7baba390de2cfeb1abcbefca346d563bf17484" -url = "https://github.com/astral-sh/uv/releases/download/0.10.12/uv-x86_64-unknown-linux-gnu.tar.gz" -url_api = "https://api.github.com/repos/astral-sh/uv/releases/assets/377491996" +checksum = "sha256:adccf40b5d1939a5e0093081ec2307ea24235adf7c2d96b122c561fa37711c46" +url = "https://github.com/astral-sh/uv/releases/download/0.10.12/uv-x86_64-unknown-linux-musl.tar.gz" provenance = "github-attestations" [tools.uv."platforms.macos-arm64"] From 1866198030d0d7f41df3b597eb47496334d47043 Mon Sep 17 00:00:00 2001 From: Brandon Squizzato Date: Fri, 11 Sep 2026 16:35:52 -0400 Subject: [PATCH 27/27] fix(docs): correct cert-manager hook ordering and clientCaSecretName comment Update ingress.mdx and openshift.mdx to describe Certificate resources as regular release objects with a post-install/post-upgrade Job, matching the current implementation and architecture/gateway.md. Fix values.yaml clientCaSecretName comment to state that "" disables client certificate verification, matching the helper and access-control docs. Signed-off-by: Brandon Squizzato --- deploy/helm/openshell/README.md | 2 +- deploy/helm/openshell/values.yaml | 4 ++-- docs/kubernetes/ingress.mdx | 2 +- docs/kubernetes/openshift.mdx | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 659a0f5d0b..3c32e24b67 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -294,7 +294,7 @@ discovery endpoint or its TLS CA. | server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. | | server.telemetryEnabled | bool | `true` | Enable anonymous OpenShell telemetry from the gateway and the sandbox supervisors it launches. | | server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. | -| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Only used when enableMtls is true. Set to "" to use the default client CA (from pkiInitJob or cert-manager). | +| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Only used when enableMtls is true. Set to "" to disable client certificate verification for HTTPS-only mode. | | server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. | | server.tls.enableMtls | bool | `true` | Enable mTLS client certificate authentication. When false, the gateway runs HTTPS-only without requiring client certificates (use OIDC for auth instead). Must be false when using BackendTLSPolicy because ingress proxies cannot present client certificates to the backend. | | server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). | diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index b2a1a1a8b0..432af61195 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -375,8 +375,8 @@ server: # proxies cannot present client certificates to the backend. enableMtls: true # -- K8s secret with ca.crt for client certificate verification (mTLS). - # Only used when enableMtls is true. Set to "" to use the default client CA - # (from pkiInitJob or cert-manager). + # Only used when enableMtls is true. Set to "" to disable client certificate + # verification for HTTPS-only mode. clientCaSecretName: openshell-server-client-ca # -- K8s secret mounted into sandbox pods for mTLS to the server. clientTlsSecretName: openshell-client-tls diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index 0623c6abd8..85d12fc8d9 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -175,7 +175,7 @@ helm upgrade --install openshell \ Note that `server.disableTls` is **not** set — the gateway pod continues to serve TLS — but `server.tls.enableMtls=false` disables mTLS client certificate authentication because the Gateway proxy cannot present a client certificate to the backend. The chart will fail the install if you try to enable both `grpcRoute.backendTLSPolicy.enabled=true` and `server.tls.enableMtls=true` simultaneously. The BackendTLSPolicy hostname defaults to the service FQDN, which matches the SAN on the server certificate. Use OIDC for authentication (configured via `server.oidc.issuer`). -The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are created first (as pre-install hooks with weight -30), then the certgen hook runs (weight -20) and polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap from it. This means a single `helm install` is sufficient in most cases. +The example above uses the default `pkiInitJob` for TLS, which creates the backend CA ConfigMap immediately. If using cert-manager instead (`--set certManager.enabled=true`), the Certificate resources are regular release objects, and a separate post-install/post-upgrade Job (`-certgen-backend-ca`) polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. This means a single `helm install` is sufficient in most cases. If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional. diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index e3aa48d6ab..cfae001a38 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -187,7 +187,7 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ Note that `server.disableTls` is **not** set — the gateway pod serves TLS over HTTPS without requiring client certificates. Use OIDC for authentication (see [Access Control](/kubernetes/access-control)). -**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are created as pre-install hooks (weight -30), then the certgen hook runs (weight -20) and polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases. +**Using cert-manager instead of pkiInitJob:** Add `--set certManager.enabled=true` to the install command. The default `certManager.serverDnsNames` already includes the service FQDN needed for BackendTLSPolicy validation. The Certificate resources are regular release objects, and a separate post-install/post-upgrade Job (`-certgen-backend-ca`) polls for up to 120 seconds waiting for cert-manager to issue the server certificate, then creates the backend CA ConfigMap. A single `helm install` is sufficient in most cases. If cert-manager takes longer than 120 seconds to issue certificates, increase the polling timeout with `--set pkiInitJob.timeoutSeconds=`. The hook polls for exactly this many seconds. For example, `timeoutSeconds=180` polls for 180 seconds. By default (`pkiInitJob.failOnTimeout=true`), the install fails if the timeout is reached, providing clear feedback that the BackendTLSPolicy is non-functional.