[the-gigi-pplx]

Summary

Adds a per-client live cap: the most sandboxes with a live runtime one ECW client key may hold at once. It's built to the per-client admission RFC in PR #1072 (docs/space/rfcs/2026-10-28-substrate-independent-client-admission.md): Decision 5 ("one count interface"), Decision 6 ("how the limits compose") and the "Per-client live cap" row of "What to keep, rework or drop". It ships off: with neither new variable set, nothing is capped and no Redis call is made.

Counting (Decision 5). Substrate::live_sandboxes_for_client(ids, activating, already_pending) is a synchronous cache read:

• Kubernetes counts from a new index kept beside the sandbox-pod reflector (kubernetes/client_live_pods.rs). It maps space.pplx.net/client-id → sandbox → qualifying pods, and pending pods count; terminal and deleting pods don't, as in the network-slot index, so a pod stuck terminating on a lost node doesn't hold its client's slot. Nor does a drain's stopped re-placement pod (checkpoint stopped-materialization), which runs no VM, so a re-placed stopped sandbox counts like one whose pod ended with its VM. It's updated from the same watch events and rebuilt whole on a re-list, like the network-slot index. A cold reflector answers Unknown(PodCacheCold). It answers Unknown(PodCacheStale) from a watch error or the end of its stream until the next live event, and during a re-list until the rebuilt index is in. A watch that's merely quiet stays trusted: the watcher reconnects a watch that's idle past its timeout. While stale, the cap counts the pending set alone: a frozen index never causes a refusal, and admissions within one pending window stay under the cap.
• Direct answers Unknown(Unsupported), and the cap admits.
• The pending set (client_live_cap/live_pending.lua) records admissions the reflectors haven't seen yet, in Redis next to the existing buckets. It keeps one sorted set per client id, with per-attempt owner tokens, an id registry for unknown's count, and a sandbox → ids companion for teardown. Every key shares one hash tag. A check reads the pending members, counts the visible sandboxes not among them, then runs a reserve script at the same cutoff. The script refuses at the cap, asks for a redo when it would refuse on members the read missed (at most three passes), and refuses to decide on a cutoff older than the grace.
• An admitted activation holds a LiveReservation. A background task renews it while the activation runs. keep() on success lets it lapse after 30 s, which covers the watch delay. Dropping it on any failure removes only that attempt's owner. A confirmed delete (or a node that had no such sandbox) and a suspend, which remove the sandbox's pods, forget its members; a failed delete leaves them to lapse.

Where the cap runs (Decision 6). The CapPolicy that PR #1068 stamps on every path now decides it:

• Reserved, before any cluster token is spent:
  • a create, before it branches between an admitted claim, a pool bypass and a pool attempt. Right after a pool claim, before the adoption's label patch, the member moves to the claimed sandbox's id, so a warm hit counts once; a fallback to cold moves it back.
  • a Start or Wake of a created or stopped sandbox.
  • an implicit access or control wake of a dormant sandbox, once the auto-wake policy allows it. A request that didn't reserve, such as a mid-stream reconnect, is checked where a pod is created instead.
• ReserveAtClaim: the Kubernetes lease winner decides it. A fresh claim reserves before its charge and becomes Reserved. A takeover of a lapsed restoring lease becomes Check, after a read-only pre-check when no reusable pod is visible, so a client at its cap doesn't pay a token on every takeover retry. The reservation rides with the detached wake leader, so a disconnected caller doesn't uncount a restore that's still running.
• Check: ensure_pod_exists checks on the branch that's about to create a pod, and before the retry path's cross-node re-drive. A refusal abandons the claim it created and returns a refused charge. The charge reaches it through PodMaterializationOwnership::with_charge.
• Deferred cold charges (the access path's started recovery, and start_execution's stale-snapshot cold start) run a read-only pre-check when the reflector shows no reusable pod (ready, not being deleted) for the sandbox, so a client at its cap doesn't spend a cluster token on every retry.
• A call that reaches a live pod creates nothing and isn't checked, and the activating sandbox never counts against itself, so a running sandbox is never refused.

Refusal. A 429 with the cluster limiter's jittered Retry-After and a message naming the key and cap. It carries no cluster-backpressure cause, so the async dispatcher retries it against its attempt budget. control_runtime_client already passes a 429 from its first call through, and its fallback wake is itself checked.

Caps. There are two env defaults: SPACE_API_CLIENT_LIVE_CAP_PER_CLIENT for listed keys, and SPACE_API_CLIENT_LIVE_CAP_UNKNOWN, which every unlisted id shares. Unset or 0 means no cap, and native is never capped. The quotas flag gains a live map (unknown sets the shared cap). An entry applies only while the env default for that key is set, and a 0 entry is ignored: the flag tunes and assigns the caps the environment enabled. Naming an id in the flag moves it from unknown to its own key under the per-client default. When the flag stops resolving, its last valid value stands for 10 minutes (RFC KEY_SET_STALENESS), then the env allowlist alone applies. The key set is the env allowlist plus, while the per-client default is set, the flag's live ids and its overrides ids (the latter while the concurrency limiter isn't off). So a flag entry can't move an id out from under unknown's cap into no cap. Each decision is counted in sandbox.api.client_live_cap.decision, tagged with the check, decision, reason and resolved key. The cap counts only while the base's SPACE_ACTIVATION_CLIENT_ATTRIBUTION_ENABLED is on: with it off, pods carry no client label, and counting the pending set alone would cap a 30 s admission rate rather than live sandboxes, so every check admits. Every fail-open case has its own reason (misconfigured, store_unavailable, unsupported, pod_cache_cold, attribution_disabled), and so does the pending-only check of a stale cache (pod_cache_stale).

Docs: docs/configuration.md (variables, plus a "Live cap" section), docs/ecw.md, and the direct.rs module doc.

Validation

At this head, on a Linux devbox: workspace cargo clippy --all-targets --all-features -D warnings is clean, and the affected tests at CI's opt-level 0 (client_live_cap, client_live_pods, recovery_charge, the ensure_pod_exists cap-policy cases, the access, ports and spaced paths, cluster_admission, lifecycle and delete paths, e2b) pass, and so does the rest of pplx-space-api, pplx-space-metrics and the DB fixtures: the full workspace (11317 of 11318 unprivileged; the one failure, node-mgr's stopped_unbound_snapshot_reaches_filesystem_without_allocating, and the privileged lane's six fs-verity, contained_launcher and spaced-guest-worker failures are this devbox's environment, in crates this PR doesn't touch), against a real Redis and Postgres. With attribution off, the cap admits without counting (drop the guard): attribution_off_admits_without_counting fails. Nothing revives a lapsed member past the cap: a_lapsed_member_is_never_revived_past_the_cap fails when a reserve renews against the read's cutoff instead of now, and when renew and swap ignore the member's expiry. CI's full run, Kind smoke included, passed on an earlier head; CI on this head covers the rest.

The full-workspace and mutation results below are from an earlier head. On a Linux devbox:

• Workspace cargo clippy --all-targets --all-features -D warnings: clean. cargo arch: no required findings.
• Targeted tests at CI's opt-level 0 (client_live_cap, client_live_pods, recovery_charge, the ensure_pod_exists cap-policy cases, sandbox_pod_view, pod_reflector, cluster_admission): 83 passed, against a real Redis and Postgres.
• Full workspace through CI's entry points. run-rust-checks.sh tests-unprivileged: 10957 of 10958 passed. tests-privileged: 61 of 67 passed. Every failure is a test that fails on this kind of devbox at main for environment reasons, and CI's lanes cover them: node-mgr stopped_unbound_snapshot_reaches_filesystem_without_allocating, and in the privileged lane the fs-verity trio, two contained_launcher "unsafe ancestors" tests and spaced-guest-worker::embedding.
• Mutation checks, one per invariant the cap relies on. Each mutant was applied and the named test re-run:
  • Pending set covers admissions the reflector hasn't seen (Lua total = visible): the_pending_set_counts_admissions_the_reflector_has_not_seen fails.
  • A running sandbox is never refused (stop excluding the activating sandbox): a_running_sandbox_is_never_refused and the_activating_sandbox_and_pending_ones_never_count fail.
  • A pending, visible sandbox counts once (stop excluding pending members from the visible count): a_pending_sandbox_the_reflector_shows_counts_once fails.
  • Direct admits as unknown (Direct answers Known(0)): direct_admits_as_unknown fails.
  • ensure_pod_exists checks a Check charge (drop Check from the policy match): a_pod_create_is_checked_against_the_live_cap_by_its_cap_policy fails.
  • A failed attempt never uncounts a sibling (release drops the member regardless of owners): a_failed_attempt_never_uncounts_a_sibling fails.
  • Teardown forgets a sandbox's members (forget removes nothing): a_kept_member_lapses_after_its_ttl_or_when_forgotten fails.
  • The run also applied two more mutants, "a flag entry can't turn a cap on" and "terminal pods don't count", but their verdicts weren't captured, so they're unverified.

Not validated: a Kind or deployed smoke run, so the cap is untested end to end through ECW handlers and the dispatcher. The create and lifecycle reserve sites have no controller-level test.

Severity and risk

Severity: does not apply. This is a new guardrail, off by default.

Risk: with no cap set, the new code paths return before any Redis call. The changes outside the cap are a new reflector index on the watch path (a few map updates per pod event), the RecoveryCharge constructors now carrying the admission handle, and teardown's forget (a no-op without a cap). With a cap set, the main risk is a wrong refusal, a 429 on a create or wake under the cap. Every uncertain count fails open, and a refused attempt removes only its own pending member. To roll back, unset both variables. Nothing persistent is written outside Redis keys that expire on their own.

Stack

Stacked on PR #1068, which provides ActivationClient, the record, the pod label, the attribution switch that gates both, and the ChargeState / CapPolicy split. This PR's base is that branch. Merging waits for the RFC in PR #1072.

Not covered

• The resolver's home. The RFC puts the id → key resolver with PR [api] tag limiter decisions and reconnect metrics with the bounded client id #1058's classification, which isn't in this base. This PR resolves keys inside client_live_cap from the same inputs (env allowlist, quotas flag) and should move onto PR [api] tag limiter decisions and reconnect metrics with the bounded client id #1058's resolver once both land. Until then, a non-label allowlist entry is ignored here rather than warned about.
• The degraded key set. The flag's last valid value is kept for KEY_SET_STALENESS (10 minutes). SPACE_API_CLIENT_QUOTAS_FLAG_REQUIRED, and counting each unlisted id on its own while degraded, are resolver behavior, so they're left to that move. Past the 10 minutes, an outage falls back to the env allowlist, as the RFC specifies when the flag isn't declared required: flag-listed clients count under unknown until the flag resolves again.
• ActivationRejectionKind::ClientAtLiveCap. The kind comes with PR api: refuse a cold create with a 503 when every eligible node's network slots are held #1061, which isn't in this base. A cap refusal is a 429 without the backpressure cause, which the dispatcher already retries against the attempt budget, the behavior the RFC gives the new kind.
• Direct. Direct's start and restore paths don't call the cap. Its count is always Unknown(Unsupported), so the outcome (admit, before any Redis round trip) is the same. Closing the gap also needs a Direct point that reserves for a ReserveAtClaim charge and checks a Check one, with RecoveryCharge's live-cap methods built without the kubernetes feature. It's a row in the direct.rs gap table. Calling it there would change Direct, which belongs in its own PR.
• A cap refusal inside a restore drive. A Check refusal from ensure_pod_exists during a lease winner's wake drive runs the wake's existing failure cleanup, which reverts the restore claim, rather than a separate refused-charge release. The outcome is the same, and the refusal reaches the caller as the 429.
• The pre-check's pod lookup. The deferred-cold pre-check finds a sandbox's pod by scanning the reflector store. It runs only on recovery paths, after the pod is already missing from the access path's view. It matches the pod by sandbox id, not by execution generation, since the claim that picks the generation runs after it. While a ready pod from an earlier generation lingers, an at-cap client can spend one cluster token before ensure_pod_exists refuses it. Matching only the current generation would instead risk refusing a running sandbox.
• A disconnect while a detached activation continues. A reservation taken at entry (a create, an explicit or implicit wake, a lease restore that claims before its wake) is held by the caller's request, while the wake leader's own reservation rides with its detached task. Once the substrate call that may create a pod is next, a dropped reservation lapses after one TTL instead of releasing, so a disconnect still counts the sandbox through the watch delay. Detached work that takes longer than the TTL to create its pod goes uncounted from then until the pod reaches the reflectors.
• No metric when no cap applies. With no cap configured for a key, the check returns without a decision metric. The cluster controller runs dispatched ECW operations through its own cap, so the SPACE_API_CLIENT_LIVE_CAP_* variables have to be set there too; docs/configuration.md says so.
• Redis Cluster. The pending-set script touches keys it doesn't declare. All of them share one hash tag, so they live in one slot, which standalone Redis and same-slot cluster access allow.
• Clients already over their cap. A client can exceed its cap only after an operator lowers it, or after a flag outage moves it under unknown. Such a client can be refused a connect to a sandbox whose pod is Running but not yet ready (the pre-checks treat only a ready pod, or a Pending one the scheduler hasn't marked Unschedulable, as reusable, so a pod left on a lost node or an unschedulable pinned restore doesn't skip them). The same holds for a pod another replica created that this replica's reflector hasn't seen yet: the pre-check, and ensure_pod_exists's own check before its create, both run before the create's conflict path would find it. A client at or under its cap isn't affected.
• A long stale period, and a member that lapses mid-activation. The RFC has a stale cache fail open; here it counts pending members alone instead, so a burst still can't pass the cap within one pending window. While the index is stale, only pending members count, and a kept member lapses 20 to 30 s after its activation succeeds. So the cap bounds each pending window, not the total: over a long watch outage a client can gain up to its cap again in each window. Likewise, if renewals fail for longer than the TTL (a Redis outage or a stalled task), the member lapses and the activation runs uncounted until its pod reaches the reflector. Both fail toward admitting, as every other store failure here does.
• Pods and sandboxes from before attribution was on. A sandbox whose pods were created by an older release or with attribution off has no recorded activating client, and a pod that replaces one of its pods without an ECW request stays unlabeled. It's counted again only after an ECW create, fork, connect or resume relabels it, so turning attribution on doesn't count a client's existing sandboxes at once. docs/configuration.md says to turn attribution on first.
• A failure after the substrate call. An activation reserved at entry (a create's warm-pool adoption from its label patch until a rollback hands the pod back (a transient label failure lapses too, since it may have landed), a cold create from its substrate call, an explicit Start, Wake or Restore, or an implicit wake once its cluster token is admitted) lets its member lapse after one TTL on any later failure, instead of releasing it, because the substrate call may have created a pod the reflector hasn't shown yet and nothing else counts it. This departs from the RFC, which releases on every failure path: a failed activation counts for up to 30 s (a substrate refusal that never creates a pod included), and a retry loop over a sandbox that keeps failing keeps it counted until the retries stop. Refusals, and failures before the substrate call (the cluster token, a cold create's database writes), still release at once, so rate-limited retries don't accumulate members. Failed creates that reach the substrate can hold at most the client's cap: once they fill it, the next create is refused before its substrate call. The cost is that a repeatable failure that never creates a pod (network-slot or cancel-storm backpressure, an unresolvable placement, an apiserver that rejects the pod) turns into a live-cap 429 for up to 30 s once a client's failed attempts fill its cap. On the shared unknown cap one unlisted client's failures do that for every unlisted client. Releasing at once on failures the substrate knows came before the pod create needs that signal plumbed out of the substrate, which this PR doesn't do. a_reservation_whose_pod_may_exist_lapses_when_dropped covers the reservation side.
• An unconfirmed pod. Wherever a pod may exist after an error (a create that fails and whose follow-up read fails too, a stale-pod cleanup that fails after the read found the pod, a failed cross-node re-drive, or a failed restore wake), the reservation is kept rather than released and lapses after one TTL. A re-drive or wake error that left no pod overcounts the same way for that TTL. This departs from the RFC, which releases on every failure path: a client whose broken sandbox keeps failing to restore under a retry loop stays counted for that sandbox until the retries stop. No test drives these paths: they need an apiserver double that fails the create and the read, or the cleanup, wired to a Redis-backed cap.
• A sandbox whose pod is deleting. A deleting pod stops counting at once, so a pod stuck terminating on a lost node never holds its client's slot. The cost: while it's deleting, its sandbox counts nowhere, so if the client fills its cap in that window, a later rebuild of that sandbox (an implicit wake or reconnect) is refused at the cap, and an uncapped replacement such as an evacuation can put the client one over.
• Override ids during a flag outage. The cap reads the quotas flag's overrides ids through the concurrency limiter's own decode, so the two agree while the flag resolves. When it stops resolving, the cap keeps the last good value for KEY_SET_STALENESS (10 minutes) and the limiter drops it at once, so for that long an id listed only in overrides is its own key here and unspecified there.
• A cap configured but unable to count. The recovery pre-checks skip their pod-store scan and record read only when no cap is configured. With a cap set but attribution off or no Redis URL, they still do that work before the cap admits.
• A suspend's cleanup racing a resume. A completed suspend, like a delete, forgets the sandbox's pending members for every owner at once. A resume that reserved its slot between the suspend committing and that cleanup loses its member, and its renewer stops, so the resumed sandbox goes uncounted until its pod reaches the reflector, about one watch delay. Scoping the cleanup to the suspended activation's own reservation needs the suspend to carry its owner token.
• A watch that stalls without an error. The count trusts the reflector until a watch error, a re-list or the end of the stream. A connection that stalls silently is caught by the watcher's idle timeout (the apiserver's 290 s watch timeout plus a margin), and is trusted until then, up to about 295 s. The RFC instead marks the cache stale after POD_CACHE_STALENESS of quiet; that isn't implemented, because a quiet watch on an idle namespace is healthy, and kube-runtime doesn't surface bookmarks to the event stream as a liveness signal. Bounding it means a shorter watcher timeout or a heartbeat.
• Site-level tests. The create and lifecycle reserve sites aren't covered by a controller-level test. Their invariants are tested at the cap and substrate level (below).

AI assistance: Claude Code wrote the code, tests and this description from the RFC.

[shehan-pplx]

Reviewed by Shehan's Review Bot