Skip to content

fix(embed): join startup workers and preserve service failures - #28656

Open
XuPeng-SH wants to merge 2 commits into
matrixorigin:mainfrom
XuPeng-SH:fix/embed-startup-diagnostics
Open

fix(embed): join startup workers and preserve service failures#28656
XuPeng-SH wants to merge 2 commits into
matrixorigin:mainfrom
XuPeng-SH:fix/embed-startup-diagnostics

Conversation

@XuPeng-SH

@XuPeng-SH XuPeng-SH commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • BUG
  • Test and CI

Which issue(s) this PR fixes:

Related: #28646

What this PR does / why we need it:

Embedded cluster startup can launch CN workers concurrently, but the previous implementation stored heterogeneous startup errors in one atomic.Value. Different concrete error types can panic while being stored, and a synchronous LOG/TN failure could return while an already-launched CN worker was still starting. That lets rollback overlap startup and hides the original failure.

The startup path now owns one error slot per service, joins every launched CN worker before returning, keeps errors in service configuration order, preserves errors.Is/errors.As, and stops scheduling services after a synchronous infrastructure failure. The fix applies to initial and incremental startup. The regression tests use the existing startFn seam, so they do not start real services or add sleeps.

The error wrapper uses MatrixOne's typed internal error to satisfy the repository SCA contract. The concurrent-error oracle has been updated to assert the resulting diagnostic while retaining the cause and ordering checks.

Validation

  • Focused startup selection is non-empty and passes in normal mode.
  • TestDoStartLockedConcurrentErrors: -race -count=100, passed (1.192s).
  • TestDoStartLockedJoinsCNBeforeReturningInfrastructureFailure: -race -count=100, passed (1.194s).
  • Full pkg/embed normal: passed (49.268s).
  • Full pkg/embed race: passed (55.684s).
  • go vet ./pkg/embed/...: passed.
  • Incremental golangci-lint --new-from-rev origin/main ./pkg/embed/...: 0 issues.
  • git diff --check: passed.

The tests were run from the rebased candidate worktree at main ee0c2bf563bf27ff2c80891bb57a1de5886bf90b with a freshly built CPU CGo/native dependency set. The first vet attempt hit a full /tmp before analysis; it was rerun with disk-backed temporary storage and passed.

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@matrix-meow matrix-meow added the size/M Denotes a PR that changes [100,499] lines label Sep 11, 2026
@XuPeng-SH
XuPeng-SH force-pushed the fix/embed-startup-diagnostics branch from 89cc564 to 490e0ff Compare September 11, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working kind/test-ci size/M Denotes a PR that changes [100,499] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants