Skip to content

fix(ops): preserve resolved component context in switchover - #10568

Merged
leon-ape merged 8 commits into
mainfrom
bugfix/sharded-switchover-runtime-target
Aug 27, 2026
Merged

leon-ape merged 8 commits into
mainfrom
bugfix/sharded-switchover-runtime-target

Conversation

@weicao

@weicao weicao commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Problem

Fixes #10567.

handleSwitchover already builds the concrete target SynthesizedComponent and resolves its ComponentDefinition TemplateVars before dispatching the lifecycle action. However, opsRuntime.Switchover discarded that resolved component and built it again. Because BuildSynthesizedComponent does not resolve TemplateVars, the lifecycle action received a new synthesized component without the resolved values produced by handleSwitchover.

The confirmed root cause is the loss of the already resolved component context. Component-name recomposition from a sharding spec name is not part of this fix.

Change

  • Remove buildSynthesizedCompByCompName and the redundant third synthesized-component build.
  • Pass the already resolved *component.SynthesizedComponent directly from handleSwitchover to opsRuntime.Switchover.
  • Use that component's concrete generated name, lifecycle actions, and resolved TemplateVars when validating instances and creating the lifecycle action.
  • Keep opsRuntime.dataContext(), dataListOpts, and GetInstance so data-plane and multi-cluster behavior is unchanged.

Preserved behavior

  • OpsRequest progress and messages remain keyed by the user-facing component or sharding name.
  • The actual lifecycle action uses the concrete generated Component name.
  • Existing source/candidate validation at the action boundary remains in place.
  • Lifecycle action parameter merging keeps the shared behavior used by all actions. This PR introduces no switchover-only parameter precedence rules.

Tests

  • pkg/controller/lifecycle full suite: PASS
  • focused pkg/operations switchover suite: PASS
  • pkg/operations full suite: PASS
  • git diff --check: PASS

The switchover coverage verifies:

  • ordinary and sharding Components use the concrete generated Component for source/candidate FQDNs;
  • a ComponentDefinition ClusterName TemplateVar resolved by ResolveTemplateNEnvVars reaches the final ActionRequest;
  • existing source/candidate disappearance cases fail at the action boundary without invoking the addon.

@weicao
weicao requested review from a team and wangyelei as code owners July 6, 2026 18:29
@github-actions github-actions Bot added the size/S Denotes a PR that changes 10-29 lines. label Jul 6, 2026
@apecloud-bot

Copy link
Copy Markdown
Collaborator

Auto Cherry-pick Instructions

Usage:
  - /nopick: Not auto cherry-pick when PR merged.
  - /pick: release-x.x [release-x.x]: Auto cherry-pick to the specified branch when PR merged.

Example:
  - /nopick
  - /pick release-1.1

CLA Recheck Instructions

Usage:
  - /recheck-cla: Trigger a re-check of CLA status for this pull request.
Example:
  - /recheck-cla

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.63636% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.64%. Comparing base (fdb5d81) to head (fe268a9).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
pkg/operations/ops_runtime.go 70.00% 3 Missing ⚠️
pkg/operations/switchover.go 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10568      +/-   ##
==========================================
+ Coverage   65.57%   65.64%   +0.07%     
==========================================
  Files         510      510              
  Lines       64186    64171      -15     
==========================================
+ Hits        42088    42125      +37     
+ Misses      18354    18310      -44     
+ Partials     3744     3736       -8     
Flag Coverage Δ
unittests 65.64% <63.63%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@weicao

weicao commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

PR #10568 patch-version runtime validation completed on the Valkey CT09 scene.

Controller image gate:

  • Source commit: 34e474cf62a8cc3b8a497eadb4f2bb6562eb6865
  • Local tag: docker.io/apecloud/kubeblocks:pr10568-34e474cf6-stella
  • Built local image digest: sha256:b2d4d6a8c4bf5f4adc87b16ff3748891ffa37c7a79f4909b72c77e4a9669e696
  • Image tar sha256: a1443776aa0ff498ea37a13c4000ba06d3e4016f7d199a522f69f6df03551d04
  • Host import: node1/node2/node3 ctr -n k8s.io images ls all show the exact tag and digest above
  • Live deployment revision: 9
  • Live controller pod: kubeblocks-7877ffdfd8-qcqzn
  • Live controller identity: image docker.io/apecloud/kubeblocks:pr10568-34e474cf6-stella, imagePullPolicy=Never, container imageID/config sha256:e72f85ce419b88ed4bbec271cf1dfbc3680df2686e844b7426c3c4f226a026e4, startTime 2026-07-06T18:42:47Z, restartCount 0, node node2, Ready true

Same frozen OpsRequest repair result:

  • Preserved namespace: valkey-ct09-fixed-foreground
  • Original stuck OpsRequest: vlk-cluster-57598-sw-t6gdq
  • After rolling this PR controller image, the same OpsRequest advanced to phase: Succeed
  • Status includes message: do switchover succeed, objectKey: Switchover/shard, status: Succeed
  • The previous DoActionFailed: Component.apps.kubeblocks.io "vlk-cluster-57598-shard" not found remains only as historical event history in describe; it did not recur after the patch rollout

Minimum clean CT09 contrast:

  • Command: ./run-tests.sh -t cluster-topology -n valkey-pr10568-clean-ct09 --valkey-major 9 --chart-version 0.1.1 --timeout-ops 600 --timeout-pod 300 with VALKEY_CLUSTER_TOPOLOGY_REQUIRED=1, VALKEY_CLUSTER_TOPOLOGY_NO_CLEANUP=1, RUNNER_CLEANUP_GATE=0
  • Result: PASS: 38, FAIL: 0, SKIP: 1, exit code 0
  • CT09 clean OpsRequest: vlk-cluster-97724-sw-ghvx6 reached Succeed; runner observed requested candidate vlk-cluster-97724-shard-6lt-0 became primary
  • Namespace valkey-pr10568-clean-ct09 preserved. Events show the CT09 OpsRequest was processed successfully and then deleted by the controller
  • CT11 skip is unrelated: default BackupRepo is Ready, but no cluster BackupPolicy was found for the clean-run cluster

Evidence packet path on the test machine:
artifacts/valkey-live/pr10568-patch-version-20260707/

Packet SHA256SUMS hash:
1f5b3a5b6be5ab9b2761583cc849f639f80af94466c537205af31f4a1fa42e93

Boundary: this validates PR #10568 patch-version CT09 scope only. It is not a full Valkey r9 or release-ready result.

@leon-ape

Copy link
Copy Markdown
Collaborator

[P2] The functional fix removes the duplicate component resolution, but the replacement expands the OpsRuntime boundary to a controller-internal type.

The merge base kept component.SynthesizedComponent inside the runtime implementation. This PR changes the public OpsRuntime.Switchover contract to accept the whole pkg/controller/component.SynthesizedComponent (pkg/operations/type.go:133). That type is the apps controller's synthesis model and carries substantially more controller/workload/template state than a switchover action needs; it is not a stable Ops or lifecycle API contract. Exporting the helper/type does not make it an appropriate cross-module contract.

Passing the already resolved concrete shard is the correct direction, but please pass a minimal stable switchover/action context instead of exposing the complete synthesized controller model through the OpsRuntime interface.

@weicao
weicao marked this pull request as draft July 14, 2026 07:30
@github-actions github-actions Bot added size/M Denotes a PR that changes 30-99 lines. and removed size/S Denotes a PR that changes 10-29 lines. labels Jul 14, 2026
@weicao

weicao commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the runtime-boundary review on exact head 984982cbb7ecfa48e1fb974ad78852329a61b9a3.

The sharded target is still resolved once by the switchover handler, but OpsRuntime now receives only a stable SwitchoverActionContext: namespace, cluster name, concrete component name, lifecycle actions, and resolved template variables. It no longer imports or accepts the controller-internal component.SynthesizedComponent. The runtime continues to list the concrete shard Pods and dispatch the same lifecycle action with the resolved candidate/current identity.

TDD/gates:

  • compile RED: *opsRuntime does not implement OpsRuntime after narrowing the interface
  • focused concrete-shard switchover GREEN
  • focused race GREEN
  • full pkg/operations GREEN
  • go vet ./pkg/operations GREEN
  • git diff --check GREEN

The PR remains Draft pending exact-head CI and review verification. Runtime N=0; this is source/local contract evidence only.

@weicao

weicao commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

/nopick

@apecloud-bot apecloud-bot added the nopick Not auto cherry-pick when PR merged label Jul 14, 2026
@weicao

weicao commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Scope disclosure added: this change intentionally fixes TemplateVars propagation as well as sharded target resolution. The handler resolves variables before dispatch; the old runtime rebuild skipped that resolver, while the new action context passes the resolved map into lifecycle.New. Runtime evidence is still N=0.

@weicao
weicao marked this pull request as ready for review July 14, 2026 15:14
@weicao

weicao commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Landing-order note: #10651 fixes an observed non-idempotent switchover replay window and touches the same handleSwitchover Pending path. Please land #10651 first. After that, #10568 will rebase onto the updated main branch, resolve the shared hunk, and rerun its exact-head focused/full gates before merge. The current #10568 head remains its reviewed source state, but it is not the final merge identity.

Comment thread pkg/operations/switchover.go Outdated
Comment thread pkg/operations/ops_runtime.go Outdated
@weicao
weicao force-pushed the bugfix/sharded-switchover-runtime-target branch from 984982c to 95b4972 Compare July 20, 2026 09:54
@github-actions github-actions Bot added size/L Denotes a PR that changes 100-499 lines. and removed size/M Denotes a PR that changes 30-99 lines. labels Jul 20, 2026
Comment thread pkg/controller/lifecycle/kbagent.go Outdated
Comment thread pkg/controller/lifecycle/kbagent.go
@github-actions github-actions Bot added size/M Denotes a PR that changes 30-99 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Aug 27, 2026
@leon-ape leon-ape changed the title fix(operations): use resolved component for switchover runtime fix(operations): preserve resolved component context in switchover Aug 27, 2026
@leon-ape leon-ape changed the title fix(operations): preserve resolved component context in switchover fix(ops): preserve resolved component context in switchover Aug 27, 2026
@leon-ape

Copy link
Copy Markdown
Collaborator

/approve

@apecloud-bot apecloud-bot added the approved PR Approved Test label Aug 27, 2026
@leon-ape
leon-ape merged commit daeb55b into main Aug 27, 2026
41 checks passed
@leon-ape
leon-ape deleted the bugfix/sharded-switchover-runtime-target branch August 27, 2026 10:14
@github-actions github-actions Bot added this to the Release 1.2.0 milestone Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved PR Approved Test nopick Not auto cherry-pick when PR merged size/M Denotes a PR that changes 30-99 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Sharded switchover fails at execution on #10360-fixed controller: runtime chain re-composes Component name from sharding spec name

3 participants