[the-gigi-pplx]

Summary

The cluster API now records the environments each node publishes. A node's complete inventory fills usable_environment_refs and known_environment_refs on node_status_snapshots, and populates the runtime_environments catalog with each environment's identity, sort key, and the upgrade and memory edges every node build that can judge them agrees on. The provisioning recipe narrows to the unretired environments each node's build knows, so a cluster-wide catalog no longer has to stay under node-ctl's 32-environment recipe limit. A new distribution samples each node's recipe size from every publication, so the limit can be alerted on before it's reached. One Postgres function, node_environment_evidence(nss), defines when those arrays count as evidence, for placement and the upgrade reconciler to share.

Terms

• Known environment: an entry in a node build's compiled catalog for its architecture, published in NodeExecutionInventory.environments.known with its definition, the runtime release's sort key, and the build's certified upgrade and memory edges.
• Usable environment: a known environment the node can cold boot from verified artifacts under its startup policy (usable_environments). Evidence, not permission.
• Certification: what one node's build says about one known environment: its sort key and its edges. Stored per node in the new node_environment_certifications table.
• Edge: an upgrade edge certifies that a sandbox on one environment can be cold-boot upgraded to another; a memory edge also certifies the target can load the source's checkpoints. runtime_environments.upgrade_edges / memory_edges hold the aggregate.
• Retired: an operator set runtime_environments.retired_at. A retired environment counts as usable on no node and is in no recipe.
• Recipe: the RuntimeProvisioningRecipe GetNodeRuntimeDesiredState serves to nodes in the provisioning cohort: the environments node-ctl should install. node-ctl refuses one over MAX_RECIPE_ENVIRONMENTS (32) whole.

The problem

Nodes publish known and usable environments with sort keys and edges (PR #980), the observed-state columns and catalog table exist (PR #979), and the recipe serves the unretired catalog (PR #1001). Nothing connected them:

1. Nothing wrote usable_environment_refs, known_environment_refs or runtime_environments. Placement had no evidence to gate on, the activation-time upgrade decision had no edges, so every move off an environment that may not execute was stranded, and an enabled recipe cohort received an empty recipe.
2. runtime_environments keeps one edge set per environment. A writer that narrows it to existing ∩ reported can never add back an edge a newer build certifies, and can't recompute anything when a node upgrades or withdraws. Intersecting over every node also lets a build that doesn't know an edge's target, such as another architecture's, veto it.
3. The recipe served the whole unretired catalog to every node. The catalog only grows until an operator retires entries, and every architecture's environments count, so the 33rd row would have stopped every admitted node from installing anything. The only signal was a per-response error log once that had already happened.
4. Environment-aware placement and the upgrade reconciler, which come next, both need the same rule for when a node's arrays are evidence, and two spellings of a freshness rule drift.

How this fixes it

• Record the arrays from a complete inventory. environment_observation::publish decodes the observation in the publication's transaction. A valid one stores the known set and the usable set minus retired environments, with environment_refs_received_at. An absent observation (an old node build) or one that fails to decode clears all three and withdraws the node's certifications, so the node has no evidence. Unavailable and legacy reports, heartbeats, and older cluster-api writers leave the arrays alone, to age out by their own receipt. The status upsert writes the three columns together, as the row's CHECK requires.
• Keep per-node certifications and aggregate edges from them. The node_environment_certifications table and the node_environment_evidence function come from PR [db] add node_environment_certifications table and node_environment_evidence function #1274, which this stacks on. record_node_certifications replaces the node's rows in node_environment_certifications, adds catalog rows for new environments, and recomputes the edges of every environment whose votes moved. An edge counts while every node whose build knows both of its ends certifies it; a build that doesn't know the target abstains. A report identical to the stored one writes nothing and takes no lock, so a steady fleet doesn't touch the catalog. A change tries a transaction-scoped advisory lock, so recomputations are serialized and each sees the previous one's committed rows. A publication never waits for it: when another node's change holds it, the publication records its arrays but not its certifications, and the node's next complete inventory tries again. An environment no node certifies any more keeps its last edges, so its sandboxes keep their upgrade paths.
• Narrow the recipe to the node's build. list_unretired_within defines the recipe: the unretired environments a known set lists, or the whole catalog for a node that hasn't reported one. list_unretired_for_node applies it to the node's known_environment_refs, reading both in one read-only snapshot. Past the limit the recipe is still omitted, as before.
• Sample the recipe size where it's written. Every complete inventory calls the same list_unretired_within, in a savepoint so a failed read skips only the sample, and emits internal_api.runtime_provisioning.recipe_environments, the size of the recipe that node would get, tagged catalog:known or catalog:full, whether or not recipes are served. It's a distribution, so the fleet maximum is readable across replicas.
• One evidence predicate. PR [db] add node_environment_certifications table and node_environment_evidence function #1274's node_environment_evidence(nss) holds when the latest inventory is complete, the node is fresh by stale_after, and the arrays were received within the node's freshness window (stale_after - observed_at: two node-status intervals plus 30 seconds, at most 10 minutes, the window direct placement already applies to inventory receipts). The docs and tests here pin it to the arrays this PR writes. Callers apply nss.reachable alongside it.

The catalog row's identity columns are spelled as published executions record them (runtime alias such as fc, runtime version, kernel component version, state format), so the upgrade decision's unbound-record attribution matches them. A definition without a kernel or a state format gets no catalog row and is in no recipe, but its certification still counts as knowing the environment. The proto now states that readers reject a whole observation when any entry fails to decode, so readers must roll out before nodes publish a new entry kind. Placement and the activation-time upgrade decision don't read the new columns or call the predicate yet; that's the placement work's job.

Validation

• runtime-environment-catalog-store Postgres tests:
  • builds_that_disagree_keep_only_the_edges_every_judging_build_certifies seeds four nodes on differing builds. It checks vetoes, abstention by a build that doesn't know the target, an upgrade that restores vetoed edges, and a withdrawal that restores a veto's edge.
  • an_environment_no_node_certifies_keeps_its_last_edges, an_unchanged_report_writes_nothing, the_catalog_row_takes_the_definition_and_the_newest_sort_key, a_retired_environment_keeps_its_certifications.
  • a_change_while_another_holds_the_aggregate_waits_for_a_later_publication holds one node's veto uncommitted while another node withdraws. The withdrawal must defer without writing or waiting, and land on its next try with the veto counted.
  • evidence_needs_a_complete_inventory_a_fresh_node_and_a_fresh_receipt covers seven rows: fresh, within the window, a receipt past the window, a stale node, unavailable, legacy, and no arrays.
  • a_node_is_assigned_the_unretired_environments_it_knows.
• internal-api-server tests through the real publish path and status upsert, with environments built from the E0 fixture:
  • a_complete_inventory_records_the_sets_and_the_catalog, including the catalog row's identity columns.
  • a_retired_environment_is_known_but_never_usable.
  • an_old_build_or_an_invalid_observation_leaves_the_node_without_evidence: a node that only ran an old build, a node downgraded to one, and a future-schema observation.
  • reports_without_a_complete_inventory_leave_the_sets_to_age.
  • nodes_on_three_builds_keep_the_edges_every_judging_build_certifies.
  • a_failed_recipe_sample_doesnt_fail_the_publication plants a catalog row the parser rejects and checks that an old build's publication, whose sample reads the whole catalog, still commits.
  • a_publication_never_waits_for_the_catalog_lock holds one node's change open, checks that another node's publication still succeeds under a short lock timeout with its arrays recorded, and that its certifications land on its next publication.
  • a_node_that_reports_its_known_environments_installs_only_those and a_cluster_catalog_over_the_limit_still_serves_nodes_that_know_fewer, through GetNodeRuntimeDesiredState.
• Mutation-checked every new guard: the recipe's known-set filter, retirement out of usable, clearing on an old build or an invalid observation, withdrawing votes, preserving on unavailable reports and heartbeats, abstention, the upgrade and memory intersections, recomputing only certified sources, the advisory lock and acting on a failed try, the unchanged-report fast path, the sort-key update, the identity requirement, and each of the evidence function's three conditions. Each mutant fails at least one test above.
• Workspace clippy (ci/scripts/run-rust-checks.sh clippy) and cargo arch pass.
• Full workspace nextest through CI's entry points on a Linux build host: the unprivileged lane passed 11109 of 11110. The one failure is node-mgr's stopped_unbound_snapshot_reaches_filesystem_without_allocating, which fails on freshly provisioned hosts and passes in CI's Rust tests job. The privileged lane passed 63 of 69. The six failures are three node-runtime-core fs-verity tests (the host kernel has no fs-verity), two node-runtime-supervisor contained_launcher tests (unsafe-ancestor checks under the host's mirror path) and spaced-guest-worker::embedding (needs a private cgroup namespace). CI's privileged lane covers all of them, and none is in a crate this change touches.

Not validated: a Kind or deployed run. No node build in a deployed cluster publishes observations yet.

Dependencies

• [db] add node_environment_certifications table and node_environment_evidence function #1274 adds the node_environment_certifications table and the node_environment_evidence function. This PR is stacked on it and merges after it. PRs [db] add observed environment columns and the runtime_environments catalog #979, [execution] publish known and usable environments with sort keys and certified edges #980 and [internal-api] serve a cohort-gated runtime provisioning recipe from the environment catalog #1001 have merged.

Not covered

• An alert on internal_api.runtime_provisioning.recipe_environments. This repository defines no monitors; the deployment that enables the recipe adds one with its other alerts.
• Placement's requirement gate and the upgrade reconciler don't call node_environment_evidence yet; the placement work adopts it.
• One driver-owned version ordering across the catalog's semver ranking, the driver comparator and VersionSortKey, and reader-side validation of RuntimeInstallation.version_sort_key, which nothing stores yet.
• Deleting a node removes its certifications without recomputing the edges it voted on. The next certification change from any node that knows those environments recomputes them.

Severity and risk

Severity: medium. Without this, environment evidence, upgrade edges and recipes all stay empty, and enabling the recipe could stop every admitted node from installing once the catalog passed 32 entries.

Cluster-api replicas built before this serve the whole unretired catalog, so keep the recipe setting off until every replica runs this build.

Risk: medium. Every complete inventory now runs three small indexed reads in its publication transaction: the node's certifications, the recipe it would be assigned, and which of its known environments are retired. Unchanged arrays aren't rewritten, so the GIN-indexed column is left alone on a steady report. A build rollout writes certifications and recomputes the aggregate once per node. A node that finds the aggregate lock taken defers to its next tick rather than waiting, so publications don't queue behind each other. The recompute probes per certified edge rather than joining every pair of environments a node knows. Nothing reads the arrays for placement yet, and the recipe and the activation-time upgrade both stay behind their default-off settings. Roll back the binary first: older cluster-api writers leave the columns alone, and the evidence ages out. PR #1274 owns the migration's rollback.

Docs

docs/execution-environments.md gains "Recording environment observations", including the recipe rollout order, and its recipe and upgrade sections no longer say nothing writes the catalog. The recipe setting's entry in docs/configuration.md describes the per-node narrowing.

AI assistance: Claude Code drafted the code, tests, docs and this description.