Skip to content

feat: implement issue #747 — [Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout.yml - #748

Open
don-petry wants to merge 3 commits into
mainfrom
dev-lead/issue-747-20260716-0846
Open

feat: implement issue #747 — [Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout.yml#748
don-petry wants to merge 3 commits into
mainfrom
dev-lead/issue-747-20260716-0846

Conversation

@don-petry

@don-petry don-petry commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

User description

Closes #747

Implemented by dev-lead agent. Please review.

Summary by CodeRabbit

  • Bug Fixes

    • Canary rollouts now handle tiers without the requested workflow gracefully.
    • Missing workflows return an empty result instead of triggering retries or failures.
    • Improved cleanup during successful, early-exit, and failed rollout checks.
  • Tests

    • Added coverage confirming missing workflows complete successfully without retries or error messages.

CodeAnt-AI Description

Treat missing tier workflows as empty results during canary rollouts

What Changed

  • Tier repositories that have never run the requested workflow now return an empty result instead of failing the fleet sweep
  • Missing workflows are handled immediately without retries or failure messages
  • Added coverage to verify successful completion, no retries, and continued fail-closed behavior for genuine fetch errors

Impact

✅ Fewer failed canary fleet sweeps
✅ No unnecessary retries for missing workflows
✅ Genuine GitHub fetch failures still surface clearly

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

@don-petry
don-petry requested a review from a team as a code owner July 16, 2026 08:56
Copilot AI review requested due to automatic review settings July 16, 2026 08:56
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@don-petry, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: e3d69d29-9d48-442f-8bc1-005e1f2a1bfc

📥 Commits

Reviewing files that changed from the base of the PR and between 0687e5e and 50935ea.

📒 Files selected for processing (1)
  • tests/canary_rollout.bats
📝 Walkthrough

Walkthrough

The canary rollout helper now captures gh run list errors, treats missing workflows as successful empty results, and cleans up temporary files across exit paths. Bats coverage verifies the empty result, success status, and absence of retries.

Changes

Canary workflow handling

Layer / File(s) Summary
Handle missing workflows and validate behavior
scripts/canary-rollout.sh, tests/canary_rollout.bats
_run_json captures stderr, returns [] without retrying when the workflow is absent, removes temporary files on exit paths, and includes tests for the resulting status and invocation count.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The script and tests address the workflow-list failure case tied to issue #747 and its canary rollout instability.
Out of Scope Changes check ✅ Passed The changes stay focused on the canary rollout helper and tests, with no clear unrelated additions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title identifies issue #747 and the affected canary rollout workflow, which matches the pull request changes.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dev-lead/issue-747-20260716-0846

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the _run_json function in scripts/canary-rollout.sh to prevent fail-closed errors when a workflow has never been run on a repository. It captures gh command errors in a temporary file and returns an empty array if the workflow is not found, with corresponding tests added in tests/canary_rollout.bats. The review feedback recommends using an EXIT trap to ensure the temporary file is reliably cleaned up, rather than relying on manual deletion at multiple exit points.

Comment thread scripts/canary-rollout.sh Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses issue #747 by updating the canary rollout orchestration script’s _run_json behavior so tier repositories that don’t have a given workflow are treated as a legitimate “no data” case ([]) rather than a transient outage that triggers retries and ultimately fails the fleet sweep.

Changes:

  • Update _run_json to capture gh run list stderr and treat “could not find any workflows named …” as an immediate [] return (no retries, no fail-closed).
  • Add Bats coverage for the “workflow not present” case, including ensuring it returns immediately without retrying.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
tests/canary_rollout.bats Adds regression tests for _run_json when a workflow is not present in a tier repo (#747).
scripts/canary-rollout.sh Adjusts _run_json to detect “workflow not present” via captured stderr and return [] instead of retry/fail-closed.

Comment thread scripts/canary-rollout.sh Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
scripts/canary-rollout.sh (1)

339-344: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Preserve and log the underlying error message.

By redirecting gh's stderr to the temporary file $errf, the actual error messages (e.g., "API rate limit exceeded", "HTTP 502", "repository not found") are now swallowed and never shown in the workflow logs for genuine failures. This will make it very difficult to debug why a rollout failed.

Please output the contents of $errf before removing it, both on the final failure and during retries, so the root cause remains observable.

🛠️ Proposed fix to log the original error
     if [ "$attempt" -ge "$attempts" ]; then
+      cat "$errf" >&2
       rm -f "$errf"
       echo "::error::_run_json: failed to fetch run list for $repo (workflow=$wf) after $attempts attempt(s)" >&2
       return 1
     fi
+    cat "$errf" >&2
     echo "::warning::_run_json: transient failure fetching run list for $repo (workflow=$wf), attempt $attempt/$attempts — retrying in ${delay}s" >&2
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/canary-rollout.sh` around lines 339 - 344, Update the _run_json
failure handling to output the contents of $errf before removing it on both the
final failure path and each retry path. Preserve the existing error and warning
messages while ensuring the captured gh stderr, including the underlying API or
repository error, remains visible in workflow logs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@scripts/canary-rollout.sh`:
- Around line 339-344: Update the _run_json failure handling to output the
contents of $errf before removing it on both the final failure path and each
retry path. Preserve the existing error and warning messages while ensuring the
captured gh stderr, including the underlying API or repository error, remains
visible in workflow logs.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: deb88725-646a-4f63-acde-0db2a1add666

📥 Commits

Reviewing files that changed from the base of the PR and between 9002c28 and 0687e5e.

📒 Files selected for processing (2)
  • scripts/canary-rollout.sh
  • tests/canary_rollout.bats

coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 16, 2026
@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #748
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-07-16T09:34:20Z

@don-petry

Copy link
Copy Markdown
Contributor Author

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-07-16T09:34:20Z

@don-petry
don-petry enabled auto-merge (squash) July 16, 2026 09:04
@don-petry
don-petry disabled auto-merge July 16, 2026 10:21
@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — review-changes (applied)

Changes committed and pushed.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 16, 2026
@don-petry
don-petry disabled auto-merge July 16, 2026 10:31
@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — review-changes (applied)

Changes committed and pushed.

@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

donpetry-bot
donpetry-bot previously approved these changes Aug 3, 2026

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 238654c54ddb3b39b4080ac45390544b26197e6c
Review mode: triage-approved (single reviewer)

Summary

Test-only change: +37 lines in tests/canary_rollout.bats, adding two regression tests for _run_json when a tier repo has never run the requested workflow (gh run list exits non-zero with "could not find any workflows named …"). The tests verify the helper returns [] with exit 0, makes exactly one gh invocation (no retries), and emits no retry/failure messages. The behavioral fix in scripts/canary-rollout.sh already landed on main; after the 2026-08-03 rebase this PR carries only the regression coverage. The "Canary-rollout Tests / Lint and bats" check passing at this head SHA confirms the new tests pass against the current script. Tests use standard bats stub patterns (temp-dir gh stub prepended to PATH, call-count log) consistent with the surrounding suite.

Linked issue analysis

Closes #747 ([Fleet Monitor] canary-rollout.yml at 69.2% failure rate, CRITICAL). Root cause was _run_json treating "workflow not present" as a transient fetch failure, exhausting retries and failing the fleet sweep closed. The fix is on main and the issue is closed; this PR adds the regression tests pinning the fixed behavior, substantively completing the issue. Prior bot review feedback (EXIT-trap cleanup, mktemp portability on the script) was addressed in earlier commits — both review threads are resolved and outdated.

Findings

No blocking findings.

  • Triage assessment confirmed: test-only diff, LOW risk, no auth/secrets/workflow-security surface. New tests correctly assert fail-open only for the specific missing-workflow error while genuine fetch errors remain fail-closed (existing adjacent test).
  • Secret scanning MCP tool was not available in this environment; the gitleaks CI check passed on this commit.
  • Note: dev-lead / dispatch and dev-lead / ci-relay show CANCELLED — these are Dev-Lead agent orchestration runs superseded by concurrency when the rebase was pushed, not code-verification checks.

CI status

All code checks green at 238654c: Lint ✓, ShellCheck ✓, Canary-rollout Tests (Lint and bats) ✓, CodeQL ✓, SonarCloud Quality Gate ✓, Secret scan (gitleaks) ✓, Agent Security Scan ✓, AgentShield ✓, npm audit ✓, CodeRabbit ✓, Graphite AI Reviews ✓. Skipped: dependabot-automerge and non-applicable dependency audits. Cancelled: two Dev-Lead agent orchestration runs (concurrency artifact of the rebase push, see Findings). Mergeable; merge state BLOCKED pending required review.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

donpetry-bot
donpetry-bot previously approved these changes Aug 3, 2026

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 238654c54ddb3b39b4080ac45390544b26197e6c
Review mode: triage-approved (single reviewer)

Summary

Test-only change: +37/-0 in tests/canary_rollout.bats, adding two regression tests for _run_json covering the tier-repo-without-workflow case (gh run list exiting non-zero with "could not find any workflows named …"). The tests pin the fixed behavior: return [] with exit 0, exactly one gh invocation (verified via call-count log), and no retry or fail-closed messaging. The behavioral fix to scripts/canary-rollout.sh already landed on main; after the 2026-08-03 rebase this PR carries only the regression coverage. The "Canary-rollout Tests / Lint and bats" check passing at this head SHA confirms the tests pass against the current script. Stub patterns (temp-dir gh shim on PATH, CANARY_GH_RETRY_SLEEP=0) match the surrounding suite.

Linked issue analysis

Closes #747 ([Fleet Monitor] canary-rollout.yml at 69.2% failure rate, CRITICAL). Root cause: _run_json treated "workflow not present" as a transient fetch failure, exhausted retries, and failed the fleet sweep closed. The script fix is on main and the issue is closed; this PR completes the work by pinning the behavior with regression tests. CodeRabbit's outside-diff suggestion (log $errf contents on genuine failures) targets scripts/canary-rollout.sh, which is no longer part of this diff post-rebase — non-blocking here. Both inline review threads (gemini: EXIT-trap quoting; copilot: mktemp template portability) are resolved and outdated.

Findings

No blocking findings.

  • Triage assessment confirmed: test-only diff, LOW risk, no auth/secrets/workflow/security surface touched.
  • Tests correctly keep fail-open scoped to the specific missing-workflow stderr message; the adjacent existing test still asserts genuine fetch errors fail closed.
  • Secret-scanning MCP tool unavailable in this environment; gitleaks CI check passed at this commit.
  • Two CANCELLED dev-lead checks (dispatch/ci-relay at 02:24 UTC) are agent-orchestration runs superseded by concurrency after the rebase push; the later dispatch run succeeded. Not code-verification checks.

CI status

All code checks green at 238654c: Lint ✓, ShellCheck ✓, Canary-rollout Tests (Lint and bats) ✓, CodeQL ✓, SonarCloud Quality Gate ✓, Secret scan (gitleaks) ✓, Agent Security Scan ✓, AgentShield ✓, npm audit ✓, CodeRabbit ✓, Graphite AI Reviews ✓. Skipped: dependabot-automerge and non-applicable ecosystem audits. Cancelled: two superseded Dev-Lead orchestration runs (see Findings). Mergeable: MERGEABLE; merge state BLOCKED pending required approval.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@donpetry-bot
donpetry-bot dismissed their stale review August 3, 2026 02:37

Superseded by automated re-review at 238654c.

donpetry-bot
donpetry-bot previously approved these changes Aug 3, 2026

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 238654c54ddb3b39b4080ac45390544b26197e6c
Review mode: triage-approved (single reviewer)

Summary

Test-only PR (+37/−0, 1 file): adds two regression tests to tests/canary_rollout.bats pinning the #747 fix — _run_json must treat "could not find any workflows named …" from gh run list as a legitimate empty result ([]) rather than a transient fetch failure. Verified the corresponding handling exists in scripts/canary-rollout.sh at the reviewed commit (workflow-not-found → cache [], return 0, no retries; genuine transport errors still retry and fail closed). Tests are well-constructed: stubbed gh in an isolated PATH dir, CANARY_GH_RETRY_SLEEP=0, call-count log proving exactly one invocation, and assertions that no retry/failure messages are emitted.

Linked issue analysis

Closes #747 (Fleet Monitor CRITICAL: canary-rollout.yml at 69.2% failure rate). Root cause — a retry storm when tier repos had never run the requested workflow — was fixed on main and this rebased PR contributes the regression coverage locking in that behavior. Issue is closed; substantively addressed.

Findings

No blocking findings.

  • Risk is LOW: test-only change, no production script, workflow, or agent-config modifications.
  • Both prior review threads (gemini: EXIT-trap note; copilot: mktemp template portability) are resolved and outdated after the rebase.
  • Secret scan: run_secret_scanning MCP tool not available in this environment; gitleaks CI check is green and the diff contains no credential-like content.
  • Triage assessment confirmed correct — nothing missed.

CI status

All code CI green at 238654c54ddb3b39b4080ac45390544b26197e6c: Lint, Lint and bats (the new tests pass), ShellCheck, CodeQL, SonarCloud quality gate, Secret scan (gitleaks), AgentShield, Agent Security Scan, npm audit. A few dev-lead / dispatch / ci-relay rollup entries show CANCELLED/SKIPPED — these are superseded agent-orchestration runs (later duplicates succeeded), not code-quality gates. Merge state BLOCKED pending required review.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@donpetry-bot
donpetry-bot dismissed their stale review August 3, 2026 02:44

Superseded by automated re-review at 238654c.

donpetry-bot
donpetry-bot previously approved these changes Aug 3, 2026

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 992708570fb6f19384952465d098400070519dd7
Review mode: triage-approved (single reviewer)

Summary

Test-only PR (+37/-0, tests/canary_rollout.bats) adding two bats regression tests for #747: _run_json must treat a tier repo's missing workflow ('could not find any workflows named …') as a legitimate empty result ([]) — returned immediately, no retries, no fail-closed. The fix itself already lives in main's scripts/canary-rollout.sh; these tests lock in the behavior. CI bats run passes on the reviewed SHA.

Linked issue analysis

Closes #747 (Fleet Monitor: canary-rollout.yml at 69.2% failure rate). Root cause — workflow-not-present exits treated as transient fetch errors, exhausting retries and failing the fleet sweep — is fixed in main (scripts/canary-rollout.sh classifies the message as a permanent empty-result condition). This PR adds the regression coverage: one test asserts [] with exit 0, another asserts exactly one gh call (no retries) and no failure messages. Issue substantively addressed.

Findings

No blocking findings.

  • Test-only change; no production code, workflows, or dependencies touched.
  • Both tests correctly stub gh via PATH inside BATS_TEST_TMPDIR, set CANARY_GH_RETRY_SLEEP=0/CANARY_GH_RETRIES=3, and assert immediate empty-result behavior plus zero retries — matching the implementation in main.
  • Fail-closed behavior for genuine fetch errors remains covered by the existing adjacent test.
  • Secret-scanning MCP tool unavailable in this environment; gitleaks CI check passed (no secrets in diff — stub scripts only).

CI status

All quality checks green on 9927085: Lint, Lint and bats, ShellCheck, CodeQL, SonarCloud (quality gate passed), Secret scan (gitleaks), Agent Security Scan, AgentShield, npm audit, CodeRabbit. Two 'dev-lead' orchestration runs show CANCELLED (superseded relays, not quality gates); ecosystem audits not applicable were SKIPPED.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@donpetry-bot
donpetry-bot dismissed their stale review August 3, 2026 02:52

Superseded by automated re-review at 9927085.

@don-petry
don-petry disabled auto-merge August 3, 2026 02:53
@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning
Issues addressed: 0
- Quality Gate passed with 0 new issues, 0 hotspots — no fixes required
Files changed: tests/canary_rollout.bats (test-only additions)
Skipped (informational): 0
Status: No action required — PR meets all quality standards
```
The PR is ready. All bot findings are clean, CI is green, and no actionable issues were identified by the bot comment.

@don-petry
don-petry enabled auto-merge (squash) August 3, 2026 02:54
donpetry-bot
donpetry-bot previously approved these changes Aug 3, 2026

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 992708570fb6f19384952465d098400070519dd7
Review mode: triage-approved (single reviewer)

Summary

Test-only PR (+37/-0, tests/canary_rollout.bats) adding two bats regression tests for the #747 fix: _run_json must treat a tier repo's missing workflow ('could not find any workflows named ...') as a legitimate empty result ([]) returned immediately with exit 0 — no retries, no fail-closed. Independently verified the behavioral fix exists in main's scripts/canary-rollout.sh at the reviewed SHA (workflow-not-found classified as a permanent empty-result condition in _repo_wf_runs_cached; only genuine transport errors retry and fail closed). The 'Lint and bats' check passing at this head SHA confirms the new tests pass against the current script. Stub patterns (isolated PATH gh shim in BATS_TEST_TMPDIR, CANARY_GH_RETRY_SLEEP=0, call-count log) match the surrounding suite.

Linked issue analysis

Closes #747 ([Fleet Monitor] canary-rollout.yml at 69.2% failure rate, CRITICAL). Root cause — workflow-not-present exits treated as transient fetch errors, exhausting retries and failing the fleet sweep closed — was fixed on main; the issue is closed. This PR completes the work by pinning the fixed behavior with regression coverage: one test asserts [] with exit 0, the other asserts exactly one gh invocation and no retry/failure messages. Substantively addressed.

Findings

No blocking findings.

  • Triage assessment confirmed: LOW risk, test-only change; no production scripts, workflows, agent configs, or dependencies touched.
  • Both review threads (gemini: EXIT-trap/local-var note; copilot: mktemp template portability) are resolved and outdated after the rebase — both targeted script code no longer in this diff.
  • No unanswered human-reviewer questions; remaining comments are bot/automation status updates.
  • Secret scanning MCP tool unavailable in this environment; gitleaks CI check is green and the diff contains no secret-like content.

CI status

All code checks green: Lint and bats, ShellCheck, CodeQL, Analyze (actions), SonarCloud (quality gate passed), Secret scan (gitleaks), Agent Security Scan, AgentShield, npm audit — all SUCCESS. Ecosystem audits not applicable (SKIPPED). The CANCELLED 'dev-lead / dispatch' and 'dev-lead / ci-relay' entries are superseded agent-orchestration relays, not code checks.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@donpetry-bot
donpetry-bot dismissed their stale review August 3, 2026 02:57

Superseded by automated re-review at 9927085.

donpetry-bot
donpetry-bot previously approved these changes Aug 3, 2026

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 992708570fb6f19384952465d098400070519dd7
Review mode: triage-approved (single reviewer)

Summary

Test-only PR (+37/-0 in tests/canary_rollout.bats) adding regression coverage for issue #747: two bats tests asserting that _run_json treats a 'could not find any workflows named …' error from gh as a legitimate empty result — returning [] with exit 0, with no retries and no fail-closed error. The behavioral fix is already on main; the passing 'Lint and bats' check on this head SHA confirms the orchestrator exhibits the asserted behavior. Triage's low-risk assessment is confirmed.

Linked issue analysis

Issue #747 (Fleet Monitor CRITICAL: canary-rollout.yml at 69.2% failure rate) was caused by tier repos that never ran a given workflow being treated as fetch failures, exhausting retries and failing the fleet sweep closed. This PR locks in the fix with two targeted regression tests: (1) missing workflow returns [] with status 0, (2) the error path returns immediately after a single gh call with no retry or 'failed to fetch run list' output. The genuine-fetch-failure fail-closed test directly above the new tests remains intact, preserving fail-closed behavior for real errors. Substantively addresses the issue (now closed).

Findings

No blocking findings. Tests follow the file's existing stub-gh pattern (mktemp -d under BATS_TEST_TMPDIR, PATH prepend, CANARY_GH_RETRY_SLEEP=0), use mktemp with a template, and include a clear comment documenting the pre-#747 failure mode. Prior inline bot review threads (gemini, copilot) are resolved and outdated. Secret scanning MCP tool unavailable in this run; gitleaks CI check passed and the diff contains no secret material.

CI status

All required checks green on 9927085: Lint, Lint and bats, ShellCheck, CodeQL, Analyze (actions), SonarCloud (quality gate passed, 0 new issues), agent-shield, Agent Security Scan, gitleaks, npm audit all SUCCESS; ecosystem audits skipped (not applicable). CANCELLED dev-lead dispatch/ci-relay entries are superseded internal automation reruns with later SUCCESS/SKIPPED runs. Branch is BEHIND main but mergeable; auto-rebase automation handles that.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@donpetry-bot
donpetry-bot dismissed their stale review August 3, 2026 03:02

Superseded by automated re-review at 9927085.

@sonarqubecloud

sonarqubecloud Bot commented Aug 3, 2026

Copy link
Copy Markdown

@don-petry
don-petry disabled auto-merge August 3, 2026 03:06
@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #748
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-08-03T03:37:09Z

@don-petry

Copy link
Copy Markdown
Contributor Author

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-08-03T03:37:09Z

@don-petry
don-petry enabled auto-merge (squash) August 3, 2026 03:07
donpetry-bot
donpetry-bot previously approved these changes Aug 3, 2026

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 50935ea21a3149cd5a9b9547e6df0c266d3d7f15
Review mode: triage-approved (single reviewer)

Summary

Test-only PR (+37 lines in tests/canary_rollout.bats) adding two regression tests for the #747 fix: _run_json must treat a tier repo that has never run the requested workflow ('could not find any workflows named …') as a legitimate empty result ([]) — immediately, without retries, and without failing the fleet sweep closed. The production fix already exists on main in scripts/canary-rollout.sh (_repo_wf_runs_cached classifies that error as permanent, caches [], and never retries); these tests lock in that behavior. Triage's low-risk assessment is confirmed.

Linked issue analysis

Issue #747 (Fleet Monitor, CRITICAL): canary-rollout.yml at 69.2% failure rate, driven by retry storms when tier repos lacked the requested workflow — each miss exhausted 6 retries with backoff, blowing sweeps out to ~20 min and getting jobs cancelled. The root fix (classify missing-workflow as empty result, no retry) is on main; this PR adds the regression coverage that pins it. Substantively addresses the issue; the tests assert both the [] result (status 0) and the single-call/no-retry behavior, while genuine fetch errors still fail closed per the adjacent existing test.

Findings

No blocking findings. Verified test correctness against the implementation on main: the stub gh emits the exact stderr string the classifier matches (case-insensitive 'could not find any workflow'), _run_json passes the empty result through jq to output [] with exit 0, and the call-log test correctly proves exactly one gh invocation with no retry warnings. Tests follow the established stub/source pattern used by neighboring tests in the same file. Secret-scanning MCP tool unavailable in this environment — relied on the green gitleaks check; diff contains only test stub content, nothing secret-like.

CI status

All quality and security checks green for 50935ea: Lint, Lint and bats, ShellCheck, CodeQL, Analyze (actions), Secret scan (gitleaks), Agent Security Scan, AgentShield, SonarCloud (quality gate passed), npm audit. The CANCELLED 'dev-lead / dispatch' and 'dev-lead / ci-relay' entries are the automation bot's own superseded runs — each has a later SUCCESS/SKIPPED rerun of the same check. mergeStateStatus BLOCKED only pending this automated review.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@donpetry-bot
donpetry-bot dismissed their stale review August 3, 2026 03:11

Superseded by automated re-review at 50935ea.

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 50935ea21a3149cd5a9b9547e6df0c266d3d7f15
Review mode: triage-approved (single reviewer)

Summary

Test-only PR (+37/-0 in tests/canary_rollout.bats) adding two regression tests for _run_json's workflow-not-present handling (#747): (1) a tier repo whose 'gh run list --workflow' fails with 'could not find any workflows named …' returns [] with exit 0 instead of failing closed, and (2) that path returns immediately without consuming retries. The behavior fix itself already landed on main in scripts/canary-rollout.sh (the missing-workflow message is classified as a permanent condition returning [], with only genuine transport errors retried/fail-closed), so after rebase this PR correctly carries only the regression coverage — confirmed by the green 'Lint and bats' check running these tests against main's script.

Linked issue analysis

Closes #747 — a Fleet Monitor CRITICAL alert for canary-rollout.yml (69.2% failure rate, 9/13 runs). Root cause was a retry storm: repos that never ran a given workflow caused non-zero 'gh run list' exits that were retried with backoff and then failed the whole fleet sweep. The fix on main plus these regression tests substantively address the alert; the issue is closed.

Findings

No blocking findings. The triage assessment (low-risk) is confirmed.

  • Diff is test-only; stubs follow the file's existing pattern (mktemp -d stub dir prepended to PATH, CANARY_GH_RETRY_SLEEP=0). No production code touched.
  • Both inline review threads (gemini-code-assist, Copilot) are resolved and outdated — they referenced earlier revisions that modified scripts/canary-rollout.sh.
  • No unanswered human-reviewer questions; latest cascade review at this same SHA already approved.
  • Secret scanning MCP tool not available in this environment; gitleaks CI check passed and the diff contains only test stub content — no secrets.

CI status

All validation checks green: Lint, Lint and bats, ShellCheck, CodeQL (actions), Secret scan (gitleaks), SonarCloud (quality gate passed), Agent Security Scan, agent-shield, npm audit, pr-auto-review dispatches. CANCELLED entries ('dev-lead / dispatch', 'dev-lead / ci-relay') are superseded agent-orchestration runs with later SUCCESS/SKIPPED entries, not code-quality gates. Mergeable: MERGEABLE.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@donpetry-bot
donpetry-bot dismissed their stale review August 3, 2026 03:15

Superseded by automated re-review at 50935ea.

@don-petry
don-petry disabled auto-merge August 3, 2026 03:54
@don-petry
don-petry enabled auto-merge (squash) August 3, 2026 03:54
@don-petry
don-petry disabled auto-merge August 3, 2026 07:47
@don-petry

Copy link
Copy Markdown
Contributor Author

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

@don-petry
don-petry enabled auto-merge (squash) August 3, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout.yml

3 participants