fix(nvca): preserve MiniService spec when saving workload config - #626
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe change preserves ChangesWorkload configuration persistence
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant saveWorkloadConfig
participant KubernetesAPI
participant MiniService
saveWorkloadConfig->>KubernetesAPI: Apply payload containing spec.workloadConfig
KubernetesAPI->>MiniService: Persist targeted field with forced ownership
KubernetesAPI-->>saveWorkloadConfig: Return applied object and resource version
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/compute-plane-services/nvca/internal/miniservice/reconcile.go`:
- Around line 481-487: Add outbound tracing around the r.Client.Patch call in
the reconciliation flow, using the existing context and preserving propagation
through the patch operation. Ensure the span is ended and records an error
status when Patch returns an error, while retaining the current error-handling
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 381a9a6e-f364-47e3-8d00-ac595c1eda39
📒 Files selected for processing (4)
src/compute-plane-services/nvca/internal/miniservice/reconcile.gosrc/compute-plane-services/nvca/internal/miniservice/reconcile_test.gosrc/compute-plane-services/nvca/internal/miniservice/revision.gosrc/compute-plane-services/nvca/internal/miniservice/transport_tls.go
Serialize workload config through the MiniService compatibility wire type and persist it with a merge patch so unrelated spec fields retain their values and field ownership. Closes #625 Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
Apply workload config through a serializer-independent unstructured SSA payload so miniservice-controller owns only spec.workloadConfig. Assert that the exact payload omits unrelated spec fields, status, and resourceVersion. Refs #625 Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
Keep revision-zero MiniServices on the install path when a spec generation changes. The install path uses the current Helm configuration and can safely resume before infrastructure objects such as the utils pod exist. Refs #625 Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
8c35d15 to
309b104
Compare
Customer Summary
NVCA now reads and saves per-workload configuration without removing required MiniService fields or interrupting initial workload installation.
TL;DR
Persist
spec.workloadConfigthrough the MiniService compatibility serializer, apply only that field through a serializer-independent server-side apply payload owned byminiservice-controller, and keep revision-zero MiniServices on the initial-install path when that spec write increments generation.Additional Details
MiniServiceSpechas custom JSON compatibility logic for canonical and legacy ICMS request names. The newerWorkloadConfigfield was missing from that wire representation, so NVCA could neither serialize nor deserialize it.saveWorkloadConfigalso constructed a partial typed MiniService and force-applied it with the controller's existing field manager. The compatibility serializer emitted zero values for required fields such asspec.namespace,spec.icmsRequestName, andspec.helmChartConfig, while omitting the intended workload config. This left reconciliation terminally stuck.This change:
WorkloadConfigto the compatibility JSON representation in both directions.apiVersion,kind,metadata.name, andspec.workloadConfig.client.Apply,FieldOwner("miniservice-controller"), and deliberate force ownership while omittingresourceVersion, so unrelated concurrent spec updates are outside the patch and do not cause whole-object conflicts.spec.workloadConfig: nullwhen the rendered control ConfigMap is removed, explicitly clearing the previously owned value while leaving unrelated fields untouched.Installingat revision 0, even if the workload-config write increments generation.Why serializer-independent targeted SSA
A typed partial
MiniServiceis not a targeted payload in this codebase: its customMiniServiceSpec.MarshalJSONmethod serializes zero-valued namespace, request-name, and Helm fields. Merely addingWorkloadConfigto that serializer would persist the flag but would still send those unrelated zero values.A JSON merge patch could safely update only
spec.workloadConfig, but it would replace the original declarative ownership model. Building an unstructured apply object preserves the intended Kubernetes server-side apply contract:miniservice-controllerdeclares ownership only ofspec.workloadConfig, force ownership resolves conflicts on that controller-owned field, and unrelated spec fields remain managed independently. The payload intentionally omitsresourceVersionso field-level SSA semantics, rather than whole-object optimistic locking, govern concurrency. See Kubernetes Server-Side Apply.Runtime sequence and failure boundary
The development BYOC test exercised this sequence:
nvcfWorkloadConfig.featureFlags.StatusByWorkerReadiness=true.nvcf-workload-configcontrol ConfigMap plus the workload objects.MiniService.spec.workloadConfigbefore applying the remaining infrastructure and workload objects.ACTIVE, then establish a successful inference baseline.The original build failed at step 4 because the typed patch emptied required spec fields and omitted the workload config. The targeted SSA change fixed that behavior, but the next runtime build exposed a second failure at the same boundary:
spec.workloadConfigsuccessfully.Installingat revision 0.generation=2andobservedGeneration=1. With no revision history yet, it classified the change as a Helm update and incremented revision to 1.utilspod, and terminally failed withPod "utils" not foundbefore infrastructure application, baseline inference, or cordon/drain.The follow-up guard treats
Installingplus revision 0 as an unambiguous initial-install state and continues throughdoInstall. This does not discard a real Helm change during installation:doInstallreads the current spec, and its render cache key includes Helm URL, values, service name, service port, and namespace, so changed Helm input causes a fresh render.For comparison, the aggressive-mode case using the same Helm workload and cluster capacity completed initial deployment, passed baseline inference, cordoned and drained a decode node, and transitioned the function to
DEGRADEDafter the expected delayed health window. That isolates the early failure to the worker-readiness workload-config lifecycle.For the Reviewer
Please focus on:
pkg/apis/nvca/v1alpha1/miniservice_json.gofor compatibility serialization completeness.internal/miniservice/reconcile.gofor the serializer-independent targeted SSA payload and the revision-zero initial-install guard.TestSaveWorkloadConfigUsesTargetedSSAfor the exact request assertion. The complete decoded patch proves that namespace, request name, Helm configuration, status, and resourceVersion are absent.TestPrepareUpgradeIfNeeded/workload_config_generation_during_initial_install_continues_installfor the lifecycle regression. It proves the controller retains revision 0 and rendered state instead of entering the update path.For QA
Local verification completed after rebasing onto current
main:The focused regression was first observed failing because revision changed from 0 to 1 and rendered state was cleared. It passes with the lifecycle guard. The full affected package run passed with Kubernetes 1.34.1 envtest assets, and changed-code lint reported
0 issues.After rebasing onto the workload-update coverage in current
main,TestReconcile_UpdateWorkloadConfig_ConfigRemovedcaught an omitted-field SSA payload that retained the old flag. The payload now sends an explicit null when configuration is removed; the affected Go packages and the Bazelminiservice_testtarget pass with that correction.Runtime QA must be repeated with a new image containing the latest commit. Verify:
spec.workloadConfig.featureFlags.StatusByWorkerReadinessistrue.miniservice-controllerownsspec.workloadConfigwithout owning unrelated spec fields inmetadata.managedFields.ACTIVE, baseline and degraded-window inference succeed, andObjectsHealthy=Falseis exposed during the fault while function availability remains active.Issues
Fixes #625
Checklist
Summary by CodeRabbit
New Features
Bug Fixes