Add a dedicated node pool for credentialed Prow jobs

[wallrj]

The periodic e2e jobs which carry Venafi credentials currently share nodes with presubmit jobs, which run unreviewed PR code in privileged (dind) pods. A container escape from a presubmit would let the attacker read the secrets of any co-scheduled pod via the node's kubelet credentials, and the shared node-local hostPath build caches give a presubmit a second path into a co-scheduled periodic. This PR gives jobs with credentials their own node pool in the untrusted cluster, tainted and labelled dedicated=credentialed-jobs, so they never share a node — or its caches — with a presubmit.

The pool nodes also get their own service account (k8s-node-credentialed) rather than reusing the worker pool's k8s-node, so the trust boundary exists at the GCP IAM layer too: a container escape onto a worker-pool node can read the node SA token from the VM metadata server, so any IAM later granted for credentialed jobs must not be attached to that shared SA. Both pools are generated from a single for_each resource so node hardening settings cannot drift between them; a moved block keeps the worker pool's existing state address.

Kubernetes' own Prow build cluster uses the same mechanism — dedicated tainted node pools which jobs opt into with a toleration:

• https://github.com/kubernetes/k8s.io/blob/33e1891fe1a3d361abf01e69bddd36dd80b80049/infra/gcp/terraform/k8s-infra-prow-build/main.tf#L106-L132
• https://github.com/kubernetes/test-infra/blob/86f26aa5bc157b43fe67911b29dc20d8a9aff3f4/config/jobs/kubernetes/sig-scalability/sig-scalability-periodic-jobs.yaml#L694-L697

Kubernetes draws its credential trust boundary at the cluster level (prow-build vs prow-build-trusted), which cert-manager already mirrors with the prow-trusted cluster; this pool adds node-level separation for the credentialed periodics that remain in the untrusted cluster.

Verified with tofu validate and tofu fmt -check -recursive (both clean); I have not run tofu plan against the real state. Note for whoever applies this: max_count = 2 raises the untrusted project's ceiling by up to 32 e2-highcpu-16 vCPUs / 300GB pd-ssd — check regional quota headroom at the first plan/apply, since a quota failure on scale-from-zero would only surface when a credentialed job first needs a node.

Companion PR: cert-manager/testing#1228 makes prowgen inject the matching nodeSelector and toleration for any periodic carrying a credential preset, at the same choke point that already strips credential presets from presubmits — so nothing relies on hand-editing individual jobs. That PR is held until this one is merged and applied.

with claude fable-5

[wallrj-cyberark]

I ran tofu plan against this branch (state refresh log trimmed, diff only shown below) to make review easier — no tofu apply was run.

Summary: 2 to add, 1 to change, 0 to destroy. All changes are confined to the cert-manager-tests-untrusted project: a new dedicated, tainted node pool (credentialed-pool-001) and its service account (k8s-node-credentialed), plus an IAM policy update adding that new service account to two existing bindings alongside the current k8s-node service account. The two "has moved" entries are pure Terraform state-address renames (existing node pools into a for_each map) with no attribute changes.

tofu plan output

  # module.cert-manager-tests-untrusted.google_project_iam_policy.project will be updated in-place
  ~ resource "google_project_iam_policy" "project" {
        id          = "cert-manager-tests-untrusted"
      ~ policy_data = jsonencode(
          ~ {
              ~ bindings = [
                    # (5 unchanged elements hidden)
                    {
                        members = [
                            "serviceAccount:service-113506928786@gcp-sa-gkehub.iam.gserviceaccount.com",
                        ]
                        role    = "roles/gkehub.serviceAgent"
                    },
                  ~ {
                      ~ members = [
                          + "serviceAccount:k8s-node-credentialed@cert-manager-tests-untrusted.iam.gserviceaccount.com",
                            "serviceAccount:k8s-node@cert-manager-tests-untrusted.iam.gserviceaccount.com",
                        ]
                        # (1 unchanged attribute hidden)
                    },
                  ~ {
                      ~ members = [
                          + "serviceAccount:k8s-node-credentialed@cert-manager-tests-untrusted.iam.gserviceaccount.com",
                            "serviceAccount:k8s-node@cert-manager-tests-untrusted.iam.gserviceaccount.com",
                        ]
                        # (1 unchanged attribute hidden)
                    },
                    {
                        members = [
                            "user:ashdavis@paloaltonetworks.com",
                            "user:ashley.davis@jetstack.io",
                            "user:mael.valais@jetstack.io",
                            "user:mvalais@paloaltonetworks.com",
                            "user:richard.wall@jetstack.io",
                            "user:riwall@paloaltonetworks.com",
                            "user:tim.ramlot@jetstack.io",
                            "user:tramlot@paloaltonetworks.com",
                        ]
                        role    = "roles/owner"
                    },
                    # (1 unchanged element hidden)
                ]
            }
        )
        # (2 unchanged attributes hidden)
    }

  # module.prow-cluster-trusted.google_container_node_pool.worker_pool has moved to module.prow-cluster-trusted.google_container_node_pool.pools["worker-pool-001"]
    resource "google_container_node_pool" "pools" {
        id                          = "projects/cert-manager-tests-trusted/locations/europe-west1-b/clusters/prow-trusted/nodePools/worker-pool-001"
        name                        = "worker-pool-001"
        # (11 unchanged attributes hidden)

        # (5 unchanged blocks hidden)
    }

  # module.prow-cluster-untrusted.google_container_node_pool.pools["credentialed-pool-001"] will be created
  + resource "google_container_node_pool" "pools" {
      + cluster                     = "prow-untrusted"
      + deletion_policy             = "DELETE"
      + id                          = (known after apply)
      + initial_node_count          = (known after apply)
      + instance_group_urls         = (known after apply)
      + location                    = "europe-west1-b"
      + managed_instance_group_urls = (known after apply)
      + max_pods_per_node           = (known after apply)
      + name                        = "credentialed-pool-001"
      + name_prefix                 = (known after apply)
      + node_count                  = (known after apply)
      + node_locations              = (known after apply)
      + operation                   = (known after apply)
      + project                     = "cert-manager-tests-untrusted"
      + version                     = (known after apply)

      + autoscaling {
          + location_policy = (known after apply)
          + max_node_count  = 2
          + min_node_count  = 0
        }

      + management {
          + auto_repair  = true
          + auto_upgrade = true
        }

      + network_config (known after apply)

      + node_config {
          + disk_size_gb       = 150
          + disk_type          = "pd-ssd"
          + effective_taints   = (known after apply)
          + gpudirect_strategy = (known after apply)
          + image_type         = "COS_CONTAINERD"
          + labels             = {
              + "dedicated" = "credentialed-jobs"
            }
          + local_ssd_count    = (known after apply)
          + logging_variant    = (known after apply)
          + machine_type       = "e2-highcpu-16"
          + metadata           = (known after apply)
          + min_cpu_platform   = (known after apply)
          + oauth_scopes       = [
              + "https://www.googleapis.com/auth/cloud-platform",
            ]
          + preemptible        = false
          + service_account    = "k8s-node-credentialed@cert-manager-tests-untrusted.iam.gserviceaccount.com"
          + spot               = false

          + boot_disk (known after apply)

          + confidential_nodes (known after apply)

          + containerd_config (known after apply)

          + gcfs_config {
              + enabled = true
            }

          + guest_accelerator (known after apply)

          + gvnic {
              + enabled = true
            }

          + kubelet_config (known after apply)

          + linux_node_config (known after apply)

          + shielded_instance_config (known after apply)

          + taint {
              + effect = "NO_SCHEDULE"
              + key    = "dedicated"
              + value  = "credentialed-jobs"
            }

          + windows_node_config (known after apply)

          + workload_metadata_config {
              + mode = "GKE_METADATA"
            }
        }

      + node_drain_config (known after apply)

      + upgrade_settings (known after apply)
    }

  # module.prow-cluster-untrusted.google_container_node_pool.worker_pool has moved to module.prow-cluster-untrusted.google_container_node_pool.pools["worker-pool-001"]
    resource "google_container_node_pool" "pools" {
        id                          = "projects/cert-manager-tests-untrusted/locations/europe-west1-b/clusters/prow-untrusted/nodePools/worker-pool-001"
        name                        = "worker-pool-001"
        # (11 unchanged attributes hidden)

        # (5 unchanged blocks hidden)
    }

  # module.prow-cluster-untrusted.google_service_account.credentialed_pool_sa[0] will be created
  + resource "google_service_account" "credentialed_pool_sa" {
      + account_id      = "k8s-node-credentialed"
      + deletion_policy = "DELETE"
      + disabled        = false
      + display_name    = "k8s node Service Account for the credentialed-jobs node pool"
      + email           = "k8s-node-credentialed@cert-manager-tests-untrusted.iam.gserviceaccount.com"
      + id              = (known after apply)
      + member          = "serviceAccount:k8s-node-credentialed@cert-manager-tests-untrusted.iam.gserviceaccount.com"
      + name            = (known after apply)
      + project         = "cert-manager-tests-untrusted"
      + unique_id       = (known after apply)
    }

Plan: 2 to add, 1 to change, 0 to destroy.

[wallrj-cyberark]

Applied. tofu apply on the plan shown above completed cleanly:

Apply complete! Resources: 2 added, 1 changed, 0 destroyed.

Post-apply checks:

• A fresh tofu plan immediately afterwards reports No changes. Your infrastructure matches the configuration. — no drift.
• The new credentialed-pool-001 node pool is RUNNING on the prow-untrusted cluster, but sitting at 0 actual nodes, since its autoscaler floor (min_node_count = 0) means nothing is created until a pod actually requests to schedule there (and the NO_SCHEDULE taint means nothing currently does, until Pin credentialed periodics to the dedicated node pool testing#1228 lands to add the matching toleration/nodeSelector).

Verification commands and output

$ gcloud container node-pools describe credentialed-pool-001 \
    --cluster=prow-untrusted --project=cert-manager-tests-untrusted --location=europe-west1-b \
    --format='value(status,initialNodeCount,autoscaling.minNodeCount,autoscaling.maxNodeCount)'
RUNNING			2

$ gcloud compute instance-groups list
NAME                                                LOCATION        SCOPE  NETWORK                     MANAGED  INSTANCES
gke-prow-untrusted-credentialed-pool--213ed192-grp  europe-west1-b  zone   k8s-prow-untrusted-network  Yes      0
gke-prow-untrusted-worker-pool-001-a511ddf2-grp     europe-west1-b  zone   k8s-prow-untrusted-network  Yes      7

No other projects (cert-manager-release, cert-manager-general, cert-manager-tests-trusted) were touched by the apply.

[inteon]

/approve
/lgtm

[cert-manager-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [inteon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment