Skip to content

fix(nvca): preserve MiniService spec when saving workload config - #626

Merged
sbaum1994 merged 5 commits into
mainfrom
sbaum/fix/miniservice-workload-config-patch
Aug 5, 2026
Merged

fix(nvca): preserve MiniService spec when saving workload config#626
sbaum1994 merged 5 commits into
mainfrom
sbaum/fix/miniservice-workload-config-patch

Conversation

@sbaum1994

@sbaum1994 sbaum1994 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Customer Summary

NVCA now reads and saves per-workload configuration without removing required MiniService fields or interrupting initial workload installation.

TL;DR

Persist spec.workloadConfig through the MiniService compatibility serializer, apply only that field through a serializer-independent server-side apply payload owned by miniservice-controller, and keep revision-zero MiniServices on the initial-install path when that spec write increments generation.

Additional Details

MiniServiceSpec has custom JSON compatibility logic for canonical and legacy ICMS request names. The newer WorkloadConfig field was missing from that wire representation, so NVCA could neither serialize nor deserialize it.

saveWorkloadConfig also constructed a partial typed MiniService and force-applied it with the controller's existing field manager. The compatibility serializer emitted zero values for required fields such as spec.namespace, spec.icmsRequestName, and spec.helmChartConfig, while omitting the intended workload config. This left reconciliation terminally stuck.

This change:

  • Adds WorkloadConfig to the compatibility JSON representation in both directions.
  • Constructs an unstructured SSA payload containing only apiVersion, kind, metadata.name, and spec.workloadConfig.
  • Retains client.Apply, FieldOwner("miniservice-controller"), and deliberate force ownership while omitting resourceVersion, so unrelated concurrent spec updates are outside the patch and do not cause whole-object conflicts.
  • Applies spec.workloadConfig: null when the rendered control ConfigMap is removed, explicitly clearing the previously owned value while leaving unrelated fields untouched.
  • Keeps a MiniService in the initial-install flow when it is Installing at revision 0, even if the workload-config write increments generation.
  • Adds regression coverage for JSON round trips, exact SSA payload contents and options, preservation of unrelated fields, and the revision-zero install race.

Why serializer-independent targeted SSA

A typed partial MiniService is not a targeted payload in this codebase: its custom MiniServiceSpec.MarshalJSON method serializes zero-valued namespace, request-name, and Helm fields. Merely adding WorkloadConfig to that serializer would persist the flag but would still send those unrelated zero values.

A JSON merge patch could safely update only spec.workloadConfig, but it would replace the original declarative ownership model. Building an unstructured apply object preserves the intended Kubernetes server-side apply contract: miniservice-controller declares ownership only of spec.workloadConfig, force ownership resolves conflicts on that controller-owned field, and unrelated spec fields remain managed independently. The payload intentionally omits resourceVersion so field-level SSA semantics, rather than whole-object optimistic locking, govern concurrency. See Kubernetes Server-Side Apply.

Runtime sequence and failure boundary

The development BYOC test exercised this sequence:

  1. Create a Helm function whose deployment configuration sets nvcfWorkloadConfig.featureFlags.StatusByWorkerReadiness=true.
  2. Render the chart successfully, including the nvcf-workload-config control ConfigMap plus the workload objects.
  3. Decode and remove that control ConfigMap from the objects that will be applied to the workload namespace.
  4. Persist the decoded flag into MiniService.spec.workloadConfig before applying the remaining infrastructure and workload objects.
  5. Wait for the function to become ACTIVE, then establish a successful inference baseline.
  6. Cordon and drain one decode-worker node and verify serving and health behavior throughout the degraded window.

The original build failed at step 4 because the typed patch emptied required spec fields and omitted the workload config. The targeted SSA change fixed that behavior, but the next runtime build exposed a second failure at the same boundary:

  1. The initial reconcile rendered the chart and applied only spec.workloadConfig successfully.
  2. That spec write incremented MiniService generation while the resource was still Installing at revision 0.
  3. The next reconcile saw generation=2 and observedGeneration=1. With no revision history yet, it classified the change as a Helm update and incremented revision to 1.
  4. The update path assumed initial infrastructure already existed, looked up the utils pod, and terminally failed with Pod "utils" not found before infrastructure application, baseline inference, or cordon/drain.

The follow-up guard treats Installing plus revision 0 as an unambiguous initial-install state and continues through doInstall. This does not discard a real Helm change during installation: doInstall reads the current spec, and its render cache key includes Helm URL, values, service name, service port, and namespace, so changed Helm input causes a fresh render.

For comparison, the aggressive-mode case using the same Helm workload and cluster capacity completed initial deployment, passed baseline inference, cordoned and drained a decode node, and transitioned the function to DEGRADED after the expected delayed health window. That isolates the early failure to the worker-readiness workload-config lifecycle.

For the Reviewer

Please focus on:

  • pkg/apis/nvca/v1alpha1/miniservice_json.go for compatibility serialization completeness.
  • internal/miniservice/reconcile.go for the serializer-independent targeted SSA payload and the revision-zero initial-install guard.
  • TestSaveWorkloadConfigUsesTargetedSSA for the exact request assertion. The complete decoded patch proves that namespace, request name, Helm configuration, status, and resourceVersion are absent.
  • TestPrepareUpgradeIfNeeded/workload_config_generation_during_initial_install_continues_install for the lifecycle regression. It proves the controller retains revision 0 and rendered state instead of entering the update path.

For QA

Local verification completed after rebasing onto current main:

go test ./internal/miniservice -run '^TestPrepareUpgradeIfNeeded$/workload_config_generation_during_initial_install_continues_install$' -count=1 -v -ldflags '...version=v25.8.0'
go test ./internal/miniservice ./pkg/apis/nvca/v1alpha1 -run 'TestPrepareUpgradeIfNeeded|TestSaveWorkloadConfigUsesTargetedSSA|TestMiniServiceSpec' -count=1 -ldflags '...version=v25.8.0'
KUBEBUILDER_ASSETS=<k8s-1.34.1-assets> go test ./internal/miniservice/... ./pkg/apis/nvca/v1alpha1/... -count=1 -ldflags '...version=v25.8.0'
GOCACHE=<isolated-cache> GOLANGCI_LINT_CACHE=<isolated-cache> golangci-lint run -c .golangci.yml --new-from-rev=origin/main

The focused regression was first observed failing because revision changed from 0 to 1 and rendered state was cleared. It passes with the lifecycle guard. The full affected package run passed with Kubernetes 1.34.1 envtest assets, and changed-code lint reported 0 issues.

After rebasing onto the workload-update coverage in current main, TestReconcile_UpdateWorkloadConfig_ConfigRemoved caught an omitted-field SSA payload that retained the old flag. The payload now sends an explicit null when configuration is removed; the affected Go packages and the Bazel miniservice_test target pass with that correction.

Runtime QA must be repeated with a new image containing the latest commit. Verify:

  • spec.workloadConfig.featureFlags.StatusByWorkerReadiness is true.
  • Existing MiniService spec fields remain unchanged.
  • miniservice-controller owns spec.workloadConfig without owning unrelated spec fields in metadata.managedFields.
  • The post-patch reconcile remains on revision 0 and proceeds through initial infrastructure and workload application.
  • The function becomes ACTIVE, baseline and degraded-window inference succeed, and ObjectsHealthy=False is exposed during the fault while function availability remains active.
  • Function cleanup removes the workload namespace and restores any cordoned node.

Issues

Fixes #625

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Summary by CodeRabbit

  • New Features

    • MiniService workload configuration, including feature flags, is now preserved across JSON operations.
    • Workload configuration updates are applied without overwriting other MiniService settings.
  • Bug Fixes

    • Initial installations now continue correctly when workload configuration changes before the first revision.
    • MiniService ownership metadata is handled consistently across related resources.

@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The change preserves WorkloadConfig during MiniService JSON conversion and applies only spec.workloadConfig. Initial installation continues when workload configuration changes. Owner references use a shared MiniService kind constant.

Changes

Workload configuration persistence

Layer / File(s) Summary
WorkloadConfig compatibility serialization
src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_json.go, src/compute-plane-services/nvca/pkg/apis/nvca/v1alpha1/miniservice_json_test.go
The compatibility JSON payload now includes WorkloadConfig. Marshal and unmarshal operations preserve workload feature flags.
Targeted workload configuration apply and installation handling
src/compute-plane-services/nvca/internal/miniservice/reconcile.go, src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go, src/compute-plane-services/nvca/internal/miniservice/revision_test.go
saveWorkloadConfig applies only spec.workloadConfig with forced ownership, deep-copies the desired configuration, and updates the resource version. Initial installation with revision zero continues without preparing a workload update. Tests cover field preservation and render-detail behavior.
Shared MiniService owner kind
src/compute-plane-services/nvca/internal/miniservice/revision.go, src/compute-plane-services/nvca/internal/miniservice/transport_tls.go
Owner reference creation and recognition now use the shared miniServiceKind constant.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant saveWorkloadConfig
  participant KubernetesAPI
  participant MiniService
  saveWorkloadConfig->>KubernetesAPI: Apply payload containing spec.workloadConfig
  KubernetesAPI->>MiniService: Persist targeted field with forced ownership
  KubernetesAPI-->>saveWorkloadConfig: Return applied object and resource version
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes address issue #625 by preserving MiniService fields, persisting WorkloadConfig, and covering the revision-zero reconciliation path.
Out of Scope Changes check ✅ Passed All changes support workload-config persistence, MiniService ownership consistency, reconciliation behavior, or regression coverage.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title follows Conventional Commits and accurately identifies the NVCA bug fix that preserves MiniService spec fields when saving workload configuration.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sbaum/fix/miniservice-workload-config-patch

Comment @coderabbitai help to get the list of available commands.

@sbaum1994
sbaum1994 marked this pull request as ready for review August 3, 2026 22:37
@sbaum1994
sbaum1994 requested a review from a team as a code owner August 3, 2026 22:37

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/compute-plane-services/nvca/internal/miniservice/reconcile.go`:
- Around line 481-487: Add outbound tracing around the r.Client.Patch call in
the reconciliation flow, using the existing context and preserving propagation
through the patch operation. Ensure the span is ended and records an error
status when Patch returns an error, while retaining the current error-handling
behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 381a9a6e-f364-47e3-8d00-ac595c1eda39

📥 Commits

Reviewing files that changed from the base of the PR and between ebe8cc1 and 8c35d15.

📒 Files selected for processing (4)
  • src/compute-plane-services/nvca/internal/miniservice/reconcile.go
  • src/compute-plane-services/nvca/internal/miniservice/reconcile_test.go
  • src/compute-plane-services/nvca/internal/miniservice/revision.go
  • src/compute-plane-services/nvca/internal/miniservice/transport_tls.go

Comment thread src/compute-plane-services/nvca/internal/miniservice/reconcile.go
Serialize workload config through the MiniService compatibility wire type and persist it with a merge patch so unrelated spec fields retain their values and field ownership.

Closes #625

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
Apply workload config through a serializer-independent unstructured SSA payload so miniservice-controller owns only spec.workloadConfig. Assert that the exact payload omits unrelated spec fields, status, and resourceVersion.

Refs #625

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
Keep revision-zero MiniServices on the install path when a spec generation changes. The install path uses the current Helm configuration and can safely resume before infrastructure objects such as the utils pod exist.

Refs #625

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
@sbaum1994
sbaum1994 force-pushed the sbaum/fix/miniservice-workload-config-patch branch from 8c35d15 to 309b104 Compare August 4, 2026 07:36
@sbaum1994
sbaum1994 enabled auto-merge August 5, 2026 07:20
@sbaum1994
sbaum1994 added this pull request to the merge queue Aug 5, 2026
Merged via the queue into main with commit c3f1503 Aug 5, 2026
18 checks passed
@sbaum1994
sbaum1994 deleted the sbaum/fix/miniservice-workload-config-patch branch August 5, 2026 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NVCA workload config persistence prunes required MiniService spec fields

2 participants