Skip to content

fix(compute-plane): update observability defaults and coverage - #648

Open
sbaum1994 wants to merge 6 commits into
mainfrom
test/bdd-observability-compute-pr3
Open

fix(compute-plane): update observability defaults and coverage#648
sbaum1994 wants to merge 6 commits into
mainfrom
test/bdd-observability-compute-pr3

Conversation

@sbaum1994

@sbaum1994 sbaum1994 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Customer Summary

Self-managed compute observability now uses published NVCA and collector versions.

TL;DR

Adds live split-cluster BDD coverage for observability.profile: compute, updates NVCA defaults to 3.1.0, and replaces the unavailable collector default with multi-architecture 0.157.9.

Additional Details

  • Installs control-plane prerequisites on ncp-local-cp, then observability and NVCA on ncp-local-compute-1.
  • Keeps version ownership out of the feature: compute-plane defaults own NVCA and the compatibility collector tag, while the source and release charts also default to 0.157.9.
  • Verifies releases, Target Allocator, NVCA health, compute monitors, collector configuration, and absent control-plane monitors.

Related work: #509, #523, and #564.

For the Reviewer

Focus on observability-compute.feature, compute-plane defaults, and the NVCA operator chart default change.

For QA

  • Live TestObservabilityCompute: 58/58 steps passed in 15m9s after destructive recreation of the split ncp-local topology.
  • NVCA source chart lint, release chart suite, vendor sync, BDD short tests/lint, observability profile test, and scoped NVCA golden comparison passed.
  • Full compute-plane golden comparison remains blocked only by existing KAI Scheduler golden drift.

No third-party dependency, NOTICE, or architecture diagram changes.

Issues

Relates to #522

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Summary by CodeRabbit

  • Improvements

    • Updated the OpenTelemetry Collector image to version 0.157.9 across compute-plane and NVCA configurations.
    • Pinned compute-plane NVCA operator and self-managed NVCA versions to 3.1.0.
    • Improved configuration handling so supplied operator and observability settings are applied consistently while defaults remain available.
  • Documentation

    • Updated image-version references and deployment guidance for the latest collector release.
  • Tests

    • Added coverage for compute-profile observability, Kubernetes context targeting, image overrides, and monitoring.

@sbaum1994
sbaum1994 requested review from a team as code owners August 4, 2026 00:24
@sbaum1994
sbaum1994 requested a review from estroz August 4, 2026 00:24
@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The change updates NVCA collector defaults, adds compute-plane image and version override handling, introduces context-aware kubectl apply command generation, and adds split-cluster observability BDD coverage.

Changes

Compute observability

Layer / File(s) Summary
NVCA collector defaults
deploy/helm/nvca-operator/..., src/compute-plane-services/nvca/...
The default OpenTelemetry Collector tag changes to 0.157.9 across chart values, schemas, templates, documentation, and image tests.
NVCA override handling
deploy/stacks/nvcf-compute-plane/...
Compute-plane values pin the operator image and NVCA version to 3.1.0. The Helmfile propagates optional image, version, and collector overrides.
Context-aware kubectl command builder
tests/bdd/dsl/...
KubectlApplyCommand validates and quotes manifest paths and adds --context when a context is provided.
Context-aware secret setup
tests/bdd/steps/...
Pull-secret application uses explicit Kubernetes contexts and keeps API keys out of command arguments.
Split-cluster observability scenario
tests/bdd/features/..., tests/bdd/godog_test.go
The BDD scenario installs and validates compute-plane observability, NVCA health, collector settings, monitoring resources, release state, feature gates, and cluster routing.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested reviewers: estroz

Sequence Diagram(s)

sequenceDiagram
  participant Godog
  participant ControlPlane
  participant Observability
  participant ComputePlane
  participant NVCA

  Godog->>ControlPlane: Configure split-cluster fixtures
  Godog->>ControlPlane: Register compute cluster
  Godog->>Observability: Install shared observability
  Godog->>ComputePlane: Install NVCA compute plane
  Godog->>NVCA: Inject credential and restart NVCA
  Godog->>ComputePlane: Verify releases and monitoring configuration
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title uses valid Conventional Commits syntax and accurately describes the observability default updates and coverage changes.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/bdd-observability-compute-pr3

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
tests/bdd/godog_test.go (1)

663-670: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Assert manifest applies for both selected contexts.

Line 663 rejects only the ambient-context form. It does not confirm that control-plane manifests use the control context and compute-plane manifests use the compute context.

A regression that sends all manifest applies to one explicit context passes this test. Count the generated apply commands and assert 16 control-plane applies and 4 compute-plane applies.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/bdd/godog_test.go` around lines 663 - 670, Update the
command-validation loop in the relevant Godog test to classify generated
manifest apply commands by explicit control-plane and compute-plane context,
then assert exactly 16 control-plane applies and 4 compute-plane applies. Retain
the existing rejection for ambient-context applies and API-key leakage, using
the test’s existing command/context symbols to distinguish the two apply groups.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh`:
- Around line 78-79: Update the regression test inputs and assertions in
observability-profile.sh to use distinct sentinel values for
global.nvcaOperator.imageTag and global.nvcaOperator.selfManaged.nvcaVersion.
Scope each check specifically to the corresponding image.tag and
selfManaged.nvcaVersion fields, so swapped mappings or values written to
unrelated matching keys fail.

In `@tests/bdd/godog_test.go`:
- Around line 640-653: The wiring test should stop requiring exact matches for
every recorded command. In the test around fakeRunner.runs, retain the existing
status == 0 assertion and replace the loop over registryLoginCommand and related
commands with a single commandRanThatContains assertion targeting the
destructive pod-delete command.

---

Nitpick comments:
In `@tests/bdd/godog_test.go`:
- Around line 663-670: Update the command-validation loop in the relevant Godog
test to classify generated manifest apply commands by explicit control-plane and
compute-plane context, then assert exactly 16 control-plane applies and 4
compute-plane applies. Retain the existing rejection for ambient-context applies
and API-key leakage, using the test’s existing command/context symbols to
distinguish the two apply groups.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e52e56b0-ac5d-4560-a1e6-12bc233ca614

📥 Commits

Reviewing files that changed from the base of the PR and between 892b704 and ffc7151.

📒 Files selected for processing (8)
  • deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
  • deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh
  • tests/bdd/dsl/kubectl.go
  • tests/bdd/dsl/kubectl_test.go
  • tests/bdd/features/observability-compute.feature
  • tests/bdd/godog_test.go
  • tests/bdd/steps/infra_steps.go
  • tests/bdd/steps/steps_test.go

Comment thread deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh Outdated
Comment thread tests/bdd/godog_test.go Outdated
Comment thread tests/bdd/dsl/kubectl.go
Comment thread tests/bdd/features/observability-compute.feature Outdated
Comment thread tests/bdd/features/observability-compute.feature Outdated
Comment thread tests/bdd/features/observability-compute.feature Outdated
Comment thread tests/bdd/features/observability-compute.feature Outdated
Comment thread tests/bdd/features/observability-compute.feature Outdated
@sbaum1994
sbaum1994 force-pushed the test/bdd-observability-compute-pr3 branch from ffc7151 to 5156232 Compare August 4, 2026 06:55
@sbaum1994 sbaum1994 changed the title test(bdd): cover compute observability profile fix(compute-plane): bump NVCA defaults and test observability Aug 4, 2026
Add explicit kube-context secret setup and optional NVCA image overrides so local split-cluster installs exercise the compute profile with published multi-arch images.

Relates to #522

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
Make the compute-plane stack own the operator and backend versions instead of overriding them in the live BDD. Remove the redundant Stargate and collector tag overrides.

Relates to #522

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
Use the latest published multi-architecture NVCF OTel collector. Keep the source and release charts, schema, tests, and compute-plane compatibility default aligned.

Relates-to: #522
Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
@sbaum1994
sbaum1994 force-pushed the test/bdd-observability-compute-pr3 branch from 5156232 to 109cc6d Compare August 5, 2026 05:19
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@sbaum1994 sbaum1994 changed the title fix(compute-plane): bump NVCA defaults and test observability fix(compute-plane): update observability defaults and coverage Aug 5, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/stacks/nvcf-compute-plane/README.md`:
- Around line 71-73: Update the README’s chart-defaults description near the
compute-plane image-tag documentation to identify chart defaults as fallback
values, not the source of tested image tags. Keep the existing statement that
compute-plane base values in 02-nvca.yaml.gotmpl provide the tested NVCA, NVCA
operator, and OTel Collector tags.

In `@tests/bdd/godog_test.go`:
- Around line 654-662: Update the command assertions in the test around the
profile-install target loop to also require the register-cluster command and an
explicit KUBECONFIG_FILE on the nvcf-compute-plane install target. Ensure every
compute operation is matched with its expected kubeconfig path so ambient
Kubernetes context cannot satisfy the test.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6ff523c5-df24-4d29-b5c6-fce01da344ce

📥 Commits

Reviewing files that changed from the base of the PR and between efc98c7 and 109cc6d.

⛔ Files ignored due to path filters (3)
  • deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/deployment.yaml is excluded by !**/testdata/**
  • deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/pre-delete-cleanup-job.yaml is excluded by !**/testdata/**
  • deploy/stacks/nvcf-compute-plane/testdata/golden/local/02-nvca.yaml-8e61e521-nvca-operator/helm-nvca-operator/templates/self-managed-nvcfbackend-cm.yaml is excluded by !**/testdata/**
📒 Files selected for processing (20)
  • deploy/helm/nvca-operator/nvca-operator/README.md
  • deploy/helm/nvca-operator/nvca-operator/templates/deployment.yaml
  • deploy/helm/nvca-operator/nvca-operator/values.schema.json
  • deploy/helm/nvca-operator/nvca-operator/values.yaml
  • deploy/helm/nvca-operator/tests/release_image_manifest_test.sh
  • deploy/helm/nvca-operator/tests/self_managed_nvca_image_reference_test.sh
  • deploy/stacks/nvcf-compute-plane/README.md
  • deploy/stacks/nvcf-compute-plane/environments/base.yaml
  • deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
  • deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh
  • src/compute-plane-services/nvca/deployments/nvca-operator/README.md
  • src/compute-plane-services/nvca/deployments/nvca-operator/templates/deployment.yaml
  • src/compute-plane-services/nvca/deployments/nvca-operator/values.schema.json
  • src/compute-plane-services/nvca/deployments/nvca-operator/values.yaml
  • tests/bdd/dsl/kubectl.go
  • tests/bdd/dsl/kubectl_test.go
  • tests/bdd/features/observability-compute.feature
  • tests/bdd/godog_test.go
  • tests/bdd/steps/infra_steps.go
  • tests/bdd/steps/steps_test.go
🚧 Files skipped from review as they are similar to previous changes (8)
  • tests/bdd/steps/steps_test.go
  • tests/bdd/dsl/kubectl.go
  • deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh
  • tests/bdd/features/observability-compute.feature
  • deploy/stacks/nvcf-compute-plane/environments/base.yaml
  • tests/bdd/dsl/kubectl_test.go
  • deploy/stacks/nvcf-compute-plane/helmfile.d/02-nvca.yaml.gotmpl
  • tests/bdd/steps/infra_steps.go

Comment thread deploy/stacks/nvcf-compute-plane/README.md
Comment thread tests/bdd/godog_test.go Outdated
"${manifest}"
)"

if [[ "${otel_collector_tag}" != "0.157.9" ]]; then

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test doesn't make sense since the version will change in future

exit 1
fi

if [[ "${backend_otel_collector_tag}" != "0.157.9" ]]; then

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as above this is a brittle test

Comment thread deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh Outdated
Comment thread deploy/stacks/nvcf-compute-plane/tests/observability-profile.sh Outdated
Comment thread tests/bdd/features/observability-compute.feature Outdated
Compare rendered collector values to their configured sources and use sentinel overrides for version routing. Keep the BDD wiring assertion focused on the destructive NVCA restart command.

Relates-to: #522
Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
Keep compute-profile coverage focused on collector enablement and the installed monitor resources.

Relates-to: #522
Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
nvcaOperator:
imageTag: "3.1.0"
selfManaged:
nvcaVersion: "3.1.0"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're working on fixing publishing now, but nvca is already on 3.1.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants