[HYPERSHELL-110] feat: per-gateway Gateway Console

[jsell-rh]

Summary

Specs Option 1 for HYPERSHELL-3: co-deploy the OpenShell dashboard ("Gateway Console") with each gateway that has a route, fronted by an oauth2-proxy sidecar. No code yet — this PR adds one sub-spec and links it from the parent.

The console is authenticated by a per-gateway confidential Keycloak client ({name}-{id}-console) whose audience and client-role mappers target the gateway client ({name}-{id}). Browser tokens therefore carry aud = {name}-{id} and hypershell.roles, so the gateway validates console traffic with the same rules it applies to the CLI, and the existing OIDC Role Bridge governs access. fullScopeAllowed = false preserves per-gateway isolation.

Settled decisions

• Confidential console client — a dedicated client per gateway; the control plane fetches its secret from Keycloak and stores it with a generated oauth2-proxy cookie secret in one per-gateway Secret. (oauth2-proxy requires a client secret even with PKCE.)
• Sidecar topology — oauth2-proxy runs beside the dashboard, upstream = localhost.
• Automatic when routed — any gateway with a route gets a console; no separate opt-in field.
• HTTP exposure — a Gateway API HTTPRoute at console-<ns>.<base-domain> on the shared Gateway's wildcard cert.

What the spec covers

Enablement, confidential client + mappers, credential Secret, Deployment (dashboard + oauth2-proxy, restricted SecurityContext, probes), Service + HTTPRoute, two NetworkPolicies, lifecycle/cleanup, atomicity/idempotency, consoleAddress (read-only data-model field + migration), RBAC (httproutes), and the shared-Gateway HTTP-listener prerequisite.

Commits

• spec per-gateway Gateway Console — the initial spec + parent index link
• prune and simplify console spec — prune ~640 → 380 lines with no loss of testable requirements

Not included

Implementation, the shared-Gateway HTTP listener (admin prerequisite), and the OpenShell dashboard image (upstream dependency).

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0979260e-a673-4491-9cdd-f22b5b5f8cce

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[jsell-rh]

Addressed Amber review 4978613478 (3 Majors)

The three findings shared one root cause: the fire-and-forget requeuer replayed the frozen original event on a goroutine that could race the reconciler, with a finite attempt budget. Rather than patch each symptom, I replaced it with the idiomatic controller pattern — a per-resource rate-limiting workqueue — which dissolves all three at the source.

Finding 1 — nil-result cancel (watcher.go:185) → d32415e
The workqueue serializes per key (a key in flight is never handed to a second worker), so a retry can't race a live event. A successful or legitimately phase-gated reconcile just Forgets the key; only a returned error requeues. No nil-based cancellation, so phase-gated/dedup no-ops never clear recovery.

Finding 2 — finite retry budget (requeue.go:132) → d32415e
Retries now use capped exponential backoff (2s..30s) indefinitely — until success, a newer spec supersedes (coalesced to latest), or the watcher context ends. No budget to exhaust mid-outage.

Finding 3 — wall-clock verify window (health.go:44) → 4f80074 + d32415e
(a) The queue coalesces to the latest spec, so a gateway un-routed after a failed provision is torn down on retry, never resurrected — removing the requeuer as a resurrection source. (b) The wall-clock window is replaced by an indefinite low-frequency absence re-check (once per routeVerifyInterval, forever), so elapsed time is never treated as proof the race drained.

Also fixed (two follow-on hazards the workqueue introduced)

• Phase-gate bypass on recovery (d32415e): the reconciler stamps phase=Provisioning before its work, and that self-write coalesces into the latest payload — a naive retry would hit the phase gate and no-op, re-stranding the gateway. A retryTransform clears the phase only on retries (NumRequeues > 0), so recovery bypasses the gate while first attempts stay gated for ordinary traffic.
• Backoff bypass via dirty-key re-adds (c1c6a9d): client-go re-queues a key marked dirty during processing immediately on Done, so the self-status watch event bypassed the backoff and spun. A per-key notBefore floor now re-defers early dirty re-adds with a cheap Get/AddAfter/Done cycle instead of re-invoking the handler.

Per-resource serialization, coalescing, indefinite retry, backoff preservation, and the phase-clear transform are covered by unit tests (internal/watcher, race-clean); the indefinite verification cadence by TestDueForVerify (internal/reconciler). go build/vet/test green. (4f80074)

[jsell-rh]

The per-key workqueue closes the three findings from the prior review: reconciliation is serialized and coalesced, retries retain capped backoff indefinitely, and teardown absence continues to be verified at low frequency without a wall-clock cutoff; the latest unused-option cleanup is behavior-neutral. The recovery path is still process-local, however, so a controller restart can permanently discard the only retry for a gateway already persisted in a phase that suppresses normal reconciliation.

Major

1. components/control-plane/internal/watcher/watcher.go:159 creates an empty in-memory queue and subscribes only to future broker events. If the controller restarts after Provisioning was persisted but before a failed reconcile completes, the queued retry disappears; WatchGateways emits no initial snapshot, an ordinary Provisioning event would be stopped by the phase gate anyway, and the health loop leaves missing provisioning work untouched. Seed recoverable gateways from current API state on startup/reconnect using a forced retry for active phases, or persist the reconciliation intent outside the process. Confidence: High (99%).

Assessment

REQUEST_CHANGES (submitted as COMMENT because this is an author self-review).

Findings Summary (ordered by severity, highest first):

1. [Major] Controller restart discards the only retry for persisted Provisioning gateways - Reconciliation / Durability (L159)

Convention Checklist (omit conventions not applicable to the diff):

Convention	Result
Reconciliation work is serialized per resource	Pass
Retry backoff survives dirty-key re-adds	Pass
Latest desired state is coalesced for retries	Pass
Retry intent survives controller restart	Fail
Cleanup verification remains convergence-based	Pass
Proper context propagation	Pass
No secrets in logs or responses	Pass
Conventional commit messages	Pass

[jsell-rh]

Restart-recovery for gateway reconciliation (review 4978760953)

Pushed 3ef6465 (control-plane) + 8127c4b (api-server) addressing the Major "Seed recovery from current gateway state".

The gateway watch stream only replays future events, so an in-process retry was lost on a controller restart — a gateway persisted at Provisioning before its workload existed stayed stranded (the health loop won't provision a missing workload, and a plain Provisioning payload hits the phase gate).

Fix — the LIST half of the standard controller LIST-then-WATCH pattern, plus the correctness subtleties a naive seed would miss:

Concern	Handling
Recover stranded gateways	seedGateways lists the inventory on every (re)connect and enqueues each; active phases (Provisioning/Degraded) are seeded phase-cleared to bypass the gate; Running is seeded verbatim (no re-provision flap)
List-watch gap	Server flushes the stream header after broker.Subscribe; watcher blocks on stream.Header() before listing
Stale overwrite	Version-aware coalescing on updated_at — newest state wins regardless of enqueue order
Delete resurrection	A pending delete is terminal; no snapshot can resurrect a resource mid-teardown
Subscription leak on reconnect	Each watch attempt runs on its own cancelable context; the reconcile queue stays on the parent context so retries survive
Pagination skew	An omitted tracked gateway is confirmed with a point GetGateway and pruned only on NotFound; a confirmed-live gateway is re-seeded with its current payload

Also resolved the outstanding earlier Major (review 4978490596): ReconcileGateway now propagates the fail-closed Gateway API error so Handle stamps Failed and the next event rebuilds a partial route.

Verification: go build/vet, go test -race (full control-plane watcher suite incl. new seed_test.go cases), gofmt clean; api-server build + go vet ./plugins/gateways/ green with the new header-handshake integration test.

[jsell-rh]

The subscription header, per-attempt cancellation, version-aware queue, and reconnect seed materially improve recovery, and the ordinary restart case now re-drives persisted work. Two ordering gaps can still lose the only forced recovery: offset pagination can omit a gateway unknown to a fresh process, and an equal-version live event can erase the seed-only phase bypass.

Major

1. components/control-plane/internal/watcher/watcher.go:304 treats one offset-paginated pass as a complete startup inventory even though this function correctly notes that those pages are not a consistent snapshot. On a fresh controller the queue has no prior key for a gateway skipped when a concurrent delete shifts later rows across a page boundary; the delete event is buffered, but the stable skipped gateway emits no event and therefore never reaches the point-read fallback. A persisted Provisioning gateway can still remain undiscovered and stranded indefinitely. Use a snapshot/cursor inventory or repeat full passes until the ID set is stable before draining the subscribed stream. Confidence: High (99%).
2. components/control-plane/internal/watcher/requeue.go:180 drops only strictly older payloads, but the forced seed is a phase-cleared clone with the same updated_at as the source Gateway. If that same-version live event was buffered during the LIST, it is accepted later and overwrites the cleared-phase payload; the first queue attempt has NumRequeues == 0, so no retry transform restores the bypass, the reconciler phase-gates the event, and Forget discards recovery. Track a sticky per-key force/recovery bit and coalesce it independently of the latest payload until an actual handler attempt consumes it. Confidence: High (99%).

Assessment

REQUEST_CHANGES (submitted as COMMENT because this is an author self-review).

Findings Summary (ordered by severity, highest first):

1. [Major] Offset pagination can omit a gateway unknown to the fresh recovery queue - Reconciliation / Durability (L304)
2. [Major] Equal-version live events can erase the forced phase-gate bypass - Reconciliation / Concurrency (L180)

Convention Checklist (omit conventions not applicable to the diff):

Convention	Result
Watch subscription established before inventory seed	Pass
Startup inventory is complete under concurrent mutation	Fail
Forced recovery survives payload coalescing	Fail
Pending deletes remain terminal	Pass
Missed-delete retries require point-read confirmation before pruning	Pass
Stream attempts cancel their broker subscriptions	Pass
Proper context propagation	Pass
No secrets in logs or responses	Pass
Conventional commit messages	Pass

[jsell-rh]

[Major] Do not treat this offset-paginated pass as a complete startup inventory. A fresh process has no prior queue key for a gateway that pagination skips. For example, a delete after page 1 shifts later rows left; page 2 can omit a stable Provisioning gateway while the collected count still reaches the new total or a short page. The subscribed stream buffers the delete, but it has no event for that skipped gateway, and the point-read fallback only examines keys already known to the queue. Use a consistent snapshot/cursor, or repeat full inventory passes until the ID set is stable before draining live events, so restart recovery cannot miss the very gateway it must discover.

Confidence: High (99%).

[jsell-rh]

[Major] Preserve forced recovery independently of payload version. The seed clears phase on a clone but retains the source Gateway's updated_at. If the corresponding live event was buffered during the LIST, this strict < comparison accepts its equal version and overwrites the phase-cleared seed. The first attempt still has NumRequeues == 0, so the retry transform does not run; the phase gate returns success and Forget removes recovery. Keep a sticky per-key force bit that is ORed across coalesced events and consumed only by a real handler attempt, rather than encoding the bypass solely in the mutable latest payload.

Confidence: High (99%).

[jsell-rh]

The subscription handshake closes the initial list-watch gap, but the client then stops reading the stream for the entire inventory seed. Because the API server EventBroker deliberately drops events for slow subscribers after its fixed 256-event buffer fills, a busy or slow seed can still permanently lose live changes.

Major

1. components/control-plane/internal/watcher/watcher.go:191 performs all inventory passes before the first Recv. The server can drain broker events into gRPC only until per-stream HTTP/2 flow control blocks; after that, the broker non-blockingly drops events once its 256-slot subscriber channel fills. Two stable ID-set passes do not detect update-only traffic, so a gateway updated after its row was read can retain a stale desired state indefinitely if that live event is dropped. Start draining the stream immediately after the header handshake, enqueueing events concurrently with the seed through the version-aware queue, and cancel/join that receiver if seeding fails. Confidence: High (99%).

Assessment

REQUEST_CHANGES (submitted as COMMENT because this is an author self-review).

Findings Summary (ordered by severity, highest first):

1. [Major] Pausing Recv during inventory seeding can overflow the lossy watch broker - Reconciliation / Durability (L191)

Convention Checklist (omit conventions not applicable to the diff):

Convention	Result
Watch subscription established before inventory seed	Pass
Live watch is drained while inventory seed runs	Fail
Event transport preserves mutations during seed	Fail
Version-aware seed/live coalescing	Pass
Stream attempts cancel their broker subscriptions	Pass
Proper context propagation	Pass
No secrets in logs or responses	Pass
Conventional commit messages	Pass

[jsell-rh]

The stable-pass inventory and equal-version sticky-force changes address the prior pagination and overwrite cases, and Recv now starts immediately after the subscription handshake so the broker buffer is drained. The new concurrency path still permits a dead-watch seed gap, can erase a pending delete, and mishandles force when versions advance, so recovery remains incomplete.

Major

1. components/control-plane/internal/watcher/watcher.go:221 runs the entire seed before observing streamErr. If Recv fails or reaches EOF during a long list pass, mutations after that failure can fall between the dead subscription and the next reconnect while the stable ID set remains unchanged. Run receive and seed under coordinated cancellation, select on both results, and cancel and join the other operation as soon as either fails. Confidence: High (99%).
2. components/control-plane/internal/watcher/watcher.go:355 unconditionally prunes after a stale knownKeys snapshot. The now-concurrent receiver can enqueue a delete after that snapshot but before GetGateway returns NotFound; prune then removes the pending delete, and the queued worker finds no payload, skipping teardown. Make pruning conditional on the current queue entry still matching the non-delete snapshot, or retain/synthesize the delete. Confidence: High (99%).
3. components/control-plane/internal/watcher/requeue.go:209 returns before setting forced when an older forced seed meets a newer live payload. If the newer payload is still Provisioning or Degraded, its normal attempt phase-gates and succeeds, so the restart recovery is forgotten. Coalesce force against the retained latest payload under the same lock, preserving it whenever that latest state still requires active-phase recovery. Confidence: High (98%).

Minor

1. components/control-plane/internal/watcher/requeue.go:216 never clears an accepted force mark when a newer Running payload replaces the active-phase seed. On a reconnect during an in-flight reconcile, the eventual Running event is therefore phase-cleared and provisioned again, despite the stated goal of avoiding a healthy re-provision flap. Recompute or clear the mark when a strictly newer payload proves forced recovery is no longer needed. Confidence: High (95%).

Assessment

REQUEST_CHANGES (submitted as COMMENT because this is an author self-review).

Findings Summary (ordered by severity, highest first):

1. [Major] Stream failure during the seed reopens the list-watch loss window - Reconciliation / Concurrency (L221)
2. [Major] Seed absence pruning can erase a concurrently queued delete - Reconciliation / Lifecycle (L355)
3. [Major] A newer active live payload can cause the older forced seed to lose recovery - Reconciliation / Durability (L209)
4. [Minor] A force mark can survive a newer Running payload and trigger redundant provisioning - Reconciliation / State (L216)

Convention Checklist (omit conventions not applicable to the diff):

Convention	Result
Watch subscription established before inventory seed	Pass
Watch stream drained during inventory seed	Pass
Stream failure atomically aborts the seed	Fail
Startup inventory stabilizes across offset pagination	Pass
Forced recovery survives equal-version coalescing	Pass
Forced recovery follows newer payload state	Fail
Pending deletes remain terminal	Fail
Seed and receiver cancellation is joined	Fail
No secrets in logs or responses	Pass
Conventional commit messages	Pass

[jsell-rh]

Commit 306bee9 closes the remaining gateway watch recovery races: receiver termination cancels and joins the seed, compare-and-prune preserves concurrent deletes, and terminal tombstones prevent a completed delete from being resurrected by stale inventory. Forced recovery now follows the retained payload across version coalescing and remains durable across health-written phases; the earlier Minor suggestion to clear force on Running is withdrawn because that phase does not prove GatewayReconciler.Handle succeeded.

Findings

No findings.

Assessment

APPROVE (submitted as COMMENT because this is an author self-review).

Findings Summary (ordered by severity, highest first):

No findings.

Convention Checklist (omit conventions not applicable to the diff):

Convention	Result
Watch subscription established before inventory seed	Pass
Watch stream drained during inventory seed	Pass
Stream failure atomically aborts the seed	Pass
Startup inventory stabilizes across offset pagination	Pass
Forced recovery survives equal- and older-version coalescing	Pass
Forced recovery survives independent health phase updates	Pass
Pending and completed deletes remain terminal	Pass
Seed and receiver cancellation is joined	Pass
Handler failures retain durable retry state	Pass
No secrets in logs or responses	Pass
Conventional commit messages	Pass

[jsell-rh]

Commits bba025f and 6a3a035 address the two concrete CI failures without changing product behavior: the Kind control-plane client now receives the existing profile mapper so its service-account identity matches the configured RBAC bypass, and the generic queue method carries a narrowly documented unused suppression. The authorization change restores the intended trusted service-account path without granting roles or weakening enforcement, while the lint-only change leaves runtime queue semantics untouched.

Findings

No findings.

Assessment

APPROVE (submitted as COMMENT because this is an author self-review).

Findings Summary (ordered by severity, highest first):

No findings.

Convention Checklist (omit conventions not applicable to the diff):

Convention	Result
Control-plane service-account token exposes the configured stable identity	Pass
Trusted service-account bypass remains exact-match and narrowly scoped	Pass
RBAC enforcement remains enabled and no bypassed method set is broadened	Pass
Attached Keycloak scope provides identity claims without granting roles	Pass
Generic queue method remains reachable through the asserted seed-sink contract	Pass
Lint suppression is local, documented, and behavior-neutral	Pass
No secrets in logs or responses	Pass
Conventional commit messages	Pass

[jsell-rh]

Head fbdc983 only merges target branch main at 66155b2; both incoming commits are already part of the target branch, and the resulting tree exactly matches the standard Git three-way merge. The effective PR change set therefore retains the previously reviewed console implementation and CI fixes without conflict-resolution drift or new PR-owned behavior.

Findings

No findings.

Assessment

APPROVE (submitted as COMMENT because this is an author self-review).

Findings Summary (ordered by severity, highest first):

No findings.

Convention Checklist (omit conventions not applicable to the diff):

Convention	Result
Target-branch refresh contains no manual conflict-resolution drift	Pass
Effective PR diff preserves reviewed console lifecycle and authorization boundaries	Pass
Previously accepted CI identity and lint fixes remain intact	Pass
Protected historical review threads remain unresolved	Pass
No secrets in logs or responses	Pass
Merge commit cleanly records current main	Pass