Skip to content

ci: make coverage UT timeouts diagnosable - #448

Merged
XuPeng-SH merged 2 commits into
matrixorigin:mainfrom
XuPeng-SH:codex/coverage-ut-timeout-diagnostics
Sep 8, 2026
Merged

ci: make coverage UT timeouts diagnosable#448
XuPeng-SH merged 2 commits into
matrixorigin:mainfrom
XuPeng-SH:codex/coverage-ut-timeout-diagnostics

Conversation

@XuPeng-SH

@XuPeng-SH XuPeng-SH commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

What failed

Fixes matrixorigin/matrixone#28419.

The external-storage coverage UT producer could spend its whole 45-minute test-step budget in an all-package CGo test and be canceled before it published a coverage profile or a useful reason. The merge job then stopped at its prerequisite guard without a coverage verdict. A raw JSON stream alone was also too noisy to diagnose quickly.

What changed

  • Keep the Python runner's controlled 60-minute test deadline, with 15-second file heartbeats and 60-second console notices. The runner records the last package/test/action, phase timing, process snapshots, SIGQUIT Go stacks, and a bounded report tail.
  • Add an independent parent-shell fallback at 70 minutes. It tracks runner ownership through a PID/process-group record, collects a final progress snapshot, terminates descendants safely, and publishes a fallback_deadline status even if the Python runner stalls.
  • Close process-ownership gaps: PID publication failures clean the already-started process group, ordinary command failures reap descendants after the group leader exits, and abnormal exits do not leave an unbounded wait.
  • Bound package discovery at 15 minutes and CGo preparation at 30 minutes. Distinguish phase timeout, phase failure, runner failure, and coverage-test failure; hard timeout KILL results are classified as timeout only when the phase consumed its configured budget.
  • Add a bounded, readable Markdown incident report: producer status/reason, event counts, failed packages/tests, error excerpts, slowest completed tests, phase timing, recent progress, and timeout/phase-failure previews. The report is line-bounded, fence-safe, and capped at 64 KiB; raw JSON and diagnostics remain available as artifacts.
  • Publish final status after the coverage profile upload and before diagnostics upload, and make the merge guard point to the readable report plus raw evidence. Add explicit setup, test, analysis, render, and artifact step limits; the coverage step has a 190-minute outer budget and the job has a 310-minute hard cap.

Validation

  • pytest -q — 56 passed, 3 xfailed.
  • actionlint 1.7.12 on all workflows.
  • YAML parsing, all workflow run blocks with bash -n, Python compilation, and git diff --check.
  • GPT-6 medium review: APPROVE after the PID/process-group, timeout-classification, budget, and report-fence fixes.

CI was not waited on per request; future deadline paths now produce a bounded status, readable report, and raw diagnostic evidence before the workflow fails.

@XuPeng-SH
XuPeng-SH merged commit 2ab1af3 into matrixorigin:main Sep 8, 2026
1 check passed
@XuPeng-SH
XuPeng-SH deleted the codex/coverage-ut-timeout-diagnostics branch September 8, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: UT coverage producer hits 60-minute timeout and cascades Coverage failure

1 participant