feat(rca): generic multi-client RCA agent plugin by ruturaj-browserstack · Pull Request #1 · browserstack/browserstack-ai-tfa-demo

ruturaj-browserstack · 2026-06-23T16:59:44Z

What this is

A portable Claude-Code/Cursor plugin that drives BrowserStack's collaborative
RCA loop (tfaRcaTurn) over all failed tests of a build — generic across
product and infra. It wraps the stable bstack MCP tools (listTestIds +
tfaRcaTurn) and adds the harness that batches RCA over a whole build, clusters
failures by signature, routes evidence requests to whatever skills/tools the
client already has, and records a per-test RCA.

Generalizes the product-/infra-coupled obs-tfa-rca skill: the loop, routing,
digest, and report core are ported; the coupling points (BrowserStack discovery,
fixed kubectl/chitragupta/bifrost, /tmp, server name) become config +
runtime capability discovery.

Companion change (separate repo, not in this PR): @browserstack/mcp-server
listTestIds gains an opt-in includeFailureDetail flag returning a trimmed
per-test failure signature — the seed for clustering with no extra probe turns.

Architecture

Three roles over the stable MCP contract:

rca-build skill (build-level orchestrator) — mandatory pre-flight GitHub
intake, discovery via listTestIds, the CSV/WAL state spine, failure-signature
clustering, build-evidence pre-compute + capability manifest, and fan-out.
ai-tfa-coordinator agent (per-test) — drives the tfaRcaTurn loop
(turn-cap, one-thread, soft-PENDING, digest-not-dump); routes each ask by
capability (no hardcoded tools); runs suspect-PR falsification.
TFA (server-side, test-level) — owns the per-test logs, authors the RCA.

Two modes, one coordinator (only the injected gap-resolver differs):

auto → workflows/rca-batch.mjs dynamic workflow (5 concurrent, no user
input; gap → "unavailable" → best-effort finalize).
interactive → subagents 5 at a time; on an evidence gap a subagent ends with
a GAP_OUTPUT (resume handles) and the orchestrator asks the user, then
re-dispatches with resume=.

Key decisions

Clustering seeds off discovery, not probe turns — signature =
normalize(category | first-error-line | file_path); representative runs the
full loop, siblings pre-seed a one-turn confirm against their own logs, with a
fall-back-to-own-loop safeguard (never blindly inherit).
Evidence routing is config + capability manifest, never hardcoded tools.
No GitHub forensics harness — references/github-evidence.md specifies the
exact evidence needed; the coordinator uses GitHub MCP → gh → degrade.
CSV is a write-ahead log (claim/heartbeat/flip + startup reaper) — in-session
resumable; pending-resume rows stay re-claimable.
Coverage stamp — TFA confidence capped by evidence coverage, so a RESOLVED
built with evidence unavailable reads as lower confidence because of the gap.

In scope: ideation #1–#5 + the v1 slice of #6 (coverage stamp), #7 (resume), #8
(conformance fixture). Deferred: #6 blast-radius digest, #8 git-forensics-MCP,
cross-session durability, Codex/Gemini orchestration parity.

Testing

51 tests pass (npm test → node --test), dependency-free.
Unit coverage: routing registry + manifest, CSV/WAL codec + claim/heartbeat/flip/
reaper + flip-guard + pending-resume resumability, signature normalization +
clustering, evidence cache, coverage band, report renderer.
Conformance fixtures replay recorded tfaRcaTurn transcripts
(resolved/blocked/pending/turn-cap) through the executable loop mirror
(lib/loop.mjs, which doubles as the sequential thin-client harness) — proving
rca capture, test_logs skip, soft-PENDING no-re-poll, turn-cap never submits a
7th turn, and the degraded no-capability path still reaches a valid terminal RCA.
workflows/rca-batch.mjs follows the documented Workflow runtime shape (meta +
pipeline/parallel/agent); it's validated via the conformance fixtures and
the unit-tested libs it relies on (the runtime DSL globals can't be unit-loaded).

Install / usage

git clone … && cd browserstack-ai-tfa-demo
cp .env.example .env   # BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY
claude --plugin-dir ./
/rca-build <build-id>

Post-Deploy Monitoring & Validation

No production/runtime impact — this is a client-side plugin (skills/agents/workflow

a bundled MCP server config), not a deployed service. Validation is manual:
run /rca-build against a known red build and confirm every failed test lands a
terminal CSV row + a per-test RCA. The two things to validate live: the sibling
one-turn-confirm cost win, and the "last green" baseline resolution (both have
safeguards so correctness doesn't depend on them).

🤖 Generated with Claude Code

…EADME Identity-only .claude-plugin/plugin.json; root .mcp.json wires the bstack MCP server (stdio); config/rca.config.json centralizes all formerly-hardcoded product/infra values (no kubectl/chitragupta/bifrost literals); /rca-build command parses build id + mode and hands off to the skill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Port the obs-tfa-rca loop decoupled: ai-tfa-coordinator drives tfaRcaTurn to a terminal RCA (turn-cap, one-thread, soft-PENDING, digest-not-dump) with the gather mechanism routed by capability (no kubectl/chitragupta/bifrost literals). lib/routing.mjs classifies each ask skip/gather/gap against the config registry + capability manifest; the gap action is the only mode fork (auto=unavailable, interactive=ask-user). references/evidence-routing.md carries the digest format and size caps verbatim. Adds sibling pre-seed one-turn-confirm hook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

SKILL.md orchestrator spec: mandatory GitHub intake ('I don't have one' → RCA-only; headless missing-input fail-fast), discovery via listTestIds(failed, includeFailureDetail), then cluster/pre-compute/fan-out/report steps. lib/csv-state.mjs is the resumable WAL spine — seed (idempotent, terminal- preserving), claim/heartbeat/flip, reaper, pendingRows — with timestamps injected (workflow-sandbox-safe) and an RFC4180 codec for multiline RCA fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…otocol lib/signature.mjs computes signature = normalize(category|error|file) off the U1 discovery payload (folds timestamps/uuids/hex/line:col/numbers), groups rows by signature, picks a deterministic representative (non-flaky, then smallest id), and leaves signal-less rows as their own singletons. references/clustering.md documents the O(causes) protocol: representative runs the full loop; siblings pre-seed a one-turn confirm against their own logs with a fall-back-to-own-loop safeguard (never blindly inherit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

buildManifest enumerates the client's discovered capabilities once into capability→{available,via}, declared to the user + TFA so no evidence is asked for that the client provably can't get. lib/evidence-cache.mjs computes the last-green→this-build delta once and caches by (repo,range,evidenceType) — fresh per-run Map, no module globals (multi-tenant-safe) — with resolveBaseline for the never-green fallback. Routes the same grounded window into every coordinator. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

workflows/rca-batch.mjs orchestrates the batch in auto mode: a pipeline over clusters dispatches ai-tfa-coordinator agents — representative full loop → siblings one-turn-confirm, no barrier between stages — with a structured RCA schema. Sandbox-correct: does no state I/O itself (orchestrator passes the clustered work-list + manifest + pre-computed build evidence via args; each coordinator agent persists its own CSV row eagerly). Gap → 'unavailable' back to TFA, no user prompt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

references/interactive-mode.md specifies the orchestrator loop: spawn ai-tfa-coordinator subagents 5 at a time; a subagent cannot pause to prompt the user, so on an evidence gap it ends early with a GAP_OUTPUT carrying resume handles (threadId+turnId); the orchestrator asks A1, then re-dispatches with resume= and the answer. Same coordinator as auto — only the gap action differs. Compact blocks not transcripts (lean main context); partial-first; auto-first/ escalate-the-residue noted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

references/github-evidence.md specifies exactly what each github ask needs (diff-since-baseline, PRs-in-window touching the failing path, blame, deploy timing) and the discovery order GitHub MCP → gh → degrade — no shipped forensics harness. Adds the adversarial falsification protocol (path overlap / deploy-state guard / direction) so only verdict:supported suspects enter related_prs; ruled-out suspects stay as disconfirming evidence. Coordinator runs it for product_code/ deploy/ci asks, reusing the pre-computed build evidence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… in U4) lib/coverage.mjs derives a per-row evidence-coverage band — TFA confidence capped by coverage (full keeps it, partial→medium, thin→low) so a RESOLVED built with evidence unavailable reads as lower confidence BECAUSE of the gap. lib/report.mjs renders the CSV to markdown: status counts + per-test table + coverage caveats, degrading missing fields to 'not available' and never crashing on an empty/partial batch. report-format.md documents the stamp, layout, and the startup reaper resume path. Blast-radius digest explicitly deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…harness lib/loop.mjs (runRcaLoop) is an executable mirror of the coordinator loop — status branching, ask routing, gap resolution, turn-cap, one-thread, soft-PENDING — driven by an injected submit(). It doubles as the D5 sequential thin-client harness. tests/conformance.test.mjs replays recorded tfaRcaTurn transcripts (resolved/blocked/pending/turn-cap fixtures) and proves: rca capture, test_logs skip, soft-PENDING no-re-poll, turn-cap never submits a 7th turn, and the degraded (no-capability auto) path still reaches a valid terminal RCA — same loop, same result. 48 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…, skip turn-cap gather Code-review fixes (suggested, non-blocking): - pending-resume removed from TERMINAL_STATES → soft-PENDING rows are now re-claimable, listed by pendingRows, and skipped by the reaper (they cleared in_flight), so the retained threadId/turnId actually drive an in-session resume instead of being stranded as a permanent non-terminal terminal. - flip() now rejects a missing/non-terminal rca_done without mutating, so a partial flip can't clear the claim yet leave the row pending (duplicate-RCA clobber). - loop checks the turn-cap BEFORE gathering, so evidence on the never-submitted final turn isn't gathered for nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Mirror the slack-mcp-plugin parallel-manifest pattern over one shared skills/ tree: add .cursor-plugin/plugin.json + .cursor-mcp.json (stdio bstack, Cursor dialect) and codex-mcp.example.toml (~/.codex/config.toml block). INTEGRATION.md documents per-host wiring — MCP config, Agent-Skills skill/agent discovery (.cursor/skills, .agents/skills, .codex/agents), the Add-to-Cursor deeplink and codex mcp add — and the one Claude-only piece (auto-mode dynamic workflow), which Cursor/Codex fill with interactive subagents or the sequential lib/loop.mjs harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

automation/: failing API cases for the rcaChat / is_mcp_driven feature (pins the jsonb 42601, setTestRca data-gate, and ownership-lockout regressions), the JUnit report (build-rca-failures.xml), and upload.sh. Uploaded as project 'RCA Feature Fencing' / build 'VRT Build' (build awswxm0t5ve7vbjnspfna4xbvjwxn92u2lwv5fw2). skills/k8s-rengg-tfa/ + bin/k8s-context.sh: the discovered k8s evidence capability — read-only obs-api context (pod health, deployed image, error-level logs, events) from the rengg-tfa namespace, error-level grep + secret redaction. Wired as the k8s discoveryHint in rca.config.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Both commands/rca-build.md (/tfa-rca:rca-build) and skills/rca-build/SKILL.md (name: rca-build) existed in the plugin — the slash invocation resolved ambiguously and returned a bare 'loaded' stub, forcing the agent to hunt for and re-read SKILL.md. Remove the redundant command (the skill is already slash-invocable and parses build id/mode/PRs in Step 0) and tighten the skill description (393 -> ~270 chars) so it registers cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…is present Auto mode is autonomous (no mid-run user input), but Step 1 said the intake was 'mandatory, both modes', so the agent prompted and STALLED waiting for intake answers even with mode=auto. Rewrite Step 1 per mode: auto gathers intake from invocation args + cheap inference (gh repo view, current branch), records any missing field as 'I don't have one' (RCA-only), shows a one-line FYI, and continues immediately — never prompting. Only interactive mode asks A1, once, upfront. Headless unchanged (build id required, rest default to none). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Plugin agents register under <plugin>:<name>, so the auto workflow's agent({agentType:'ai-tfa-coordinator'}) failed with 'agent type not found'. Use 'tfa-rca:ai-tfa-coordinator' in rca-batch.mjs (both representative + sibling dispatch) and fix the interactive-mode / agent-doc dispatch examples to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…e_all) The prior fix only updated the sibling call (different indentation); the representative agent() at the top of the pipeline still used the bare 'ai-tfa-coordinator' and would fail identically. Both now use the namespaced type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- skills/rca-build/ -> skills/factory/ (frontmatter name: factory); no command file with the same name (collision lesson, see 2af22ef) - delete the mode concept: no mode arg, no auto|interactive fork, drop references/interactive-mode.md - restructure to ONE GATE before execution: Part A = connector discovery + cheap probe validation into a connector -> valid|invalid|absent manifest (gaps recorded, never blockers); Part B = intake resolved by assumption, at most ONE consolidated question at gate close (headless never asks). After gate close the run never asks the user again. - no local report: drop references/report-format.md and config paths.reportFile; finish = glimpse table + triggerRcaReport + Test Observability UI link - PRODUCT_BUG mandate in github-evidence.md: application-bug RCA must carry culprit PR link(s) or explicitly state what was searched - server emits only NEEDS_INFO|RESOLVED: drop BLOCKED from references Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… local report - lib/loop.mjs: RESOLVED now carries {glimpse:{root_cause,failure_type, related_prs}, viewRca} — no full rca payload; drop BLOCKED (server emits only NEEDS_INFO|RESOLVED, plus soft-PENDING); drop the resolveGap injection — gaps always degrade to an 'unavailable' block (auto-only, no prompt path) - delete lib/report.mjs and .rca/rca-report*.md — the plugin never renders a local RCA report; new lib/glimpse.mjs renders the terse end-of-run glimpse table (testRunId → cluster → status → confidence one-liner) from the CSV spine - fixtures: resolved.json/pending.json to the trimmed shapes; drop blocked.json; conformance + coverage tests updated (49 green) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- ai-tfa-coordinator: drop the mode input and the interactive gap-return (GAP_OUTPUT); gaps always degrade to 'unavailable' — never a prompt. Classify only RESOLVED|PENDING|NEEDS_INFO (BLOCKED gone); consume the trimmed RESOLVED glimpse {root_cause, failure_type, related_prs} + viewRca and pass it through verbatim; RCA_OUTPUT gains failure_type + view_rca, drops possible_fix/BLOCKED - mandatory culprit-PR hunt for PRODUCT_BUG / application bugs: feed PR link(s) to TFA so dashboard related_prs populates; no PR by turn cap → explicit 'no culprit PR identified after <searched>' + recorded gap - workflows/rca-batch.mjs: single autonomous path (no mode arg), schema matches the new RCA_OUTPUT, PRODUCT_BUG mandate in the shared prompt Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…CKED residue - README: /factory usage, the one-gate two-part flow (validated connector manifest + assumption-first intake, at most one consolidated question), autonomous execution, glimpse + triggerRcaReport + Test Observability UI link as the only outputs - INTEGRATION: factory skill naming, triggerRcaReport in the MCP core row, no-local-report finish on every host - automation/README: demo invocation is now /factory <build-id> - lib/routing.mjs comments: gaps degrade to 'unavailable', no resolveGap Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…c-generator skill Revert only the rename from the single-gate rework (name collision with the user-level /factory skill); all auto-only, single-gate, glimpse, and PRODUCT_BUG-mandate behavior stays. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…eDir) Two fixes to the state spine's location: 1. The build id is now IN the filename (rca-state.<buildId>.csv) — previously a fixed .rca/rca-state.csv meant a run on build B would collide with and 'resume' into build A's state. 2. Default directory is OS temp (<tmpdir>/bstack-rca/), not the invoking workspace — a background harness must not pollute the repo it runs from. paths.stateDir remains an override for retained artifacts (e.g. CI). Resume-safety is preserved per build (same id -> same path). Hostile ids are sanitized. 4 new tests; 53/53 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…les/, scripts/ - k8s-rengg-tfa is self-contained: bin/k8s-context.sh -> skills/k8s-rengg-tfa/ scripts/ (carries the pending redaction hardening for auth headers) - rca-build gains templates/ (suspect-packet, evidence-block, gate-summary) as the canonical fillable formats — references now point at them instead of embedding drift-prone copies — and examples/sample-run.md (fictional, matches the recorded-turn fixtures): gate summary, answered NEEDS_INFO turn with supported + ruled-out suspects, terminal glimpse output Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ai_report/aitfa tabs The viewReport/viewRca destination is the build's AI-TFA report: <base>/builds/<buildUuid>?tab=ai_report&subTab=aitfa on automation.browserstack.com (designer-confirmed; was an observability.browserstack.com assumption). Fixture now carries the guidance string the bridge actually emits on RESOLVED turns — closes the fixture-drift review finding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Environment-specific harness removed; k8s stays a generic gate-probed capability (discoveryHints now empty, like logs/metrics). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…runtime the user has The harness no longer assumes Kubernetes. 'infra' is the capability (runtime state of the service); k8s/EKS, ECS, docker, Nomad, VMs, PM2 are all just possible connectors behind it — the gate probes what actually exists and records the kind in the manifest (via). The k8s evidenceType remains a routing alias into infra so TFA asks spelled 'k8s' keep working. kubectl is one probe on a menu, not the definition of the capability. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…; note prod default base URL Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…beta.1 Pinned across all three client wirings (.mcp.json, .cursor-mcp.json, codex example) + INTEGRATION.md. Codex example env aligned with the prod-default base URL (staging override optional). Version verified present on the public npm registry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…erence The automation/ demo (failing rcaChat API cases + upload.sh, named observability-api as 'the product repo') was env-specific bait: the gate latched onto the only repo named in a workspace doc and carried its PRs into the manifest — even for a build whose failures were self-healing tests, wholly unrelated to observability-api. - Delete automation/ (tests, upload.sh, README, fixture xml). - Gate Part B now REQUIRES product-repo corroboration against the failure signatures; a doc-sourced repo is a weak hint, never an assumption — if it doesn't match the failures it's recorded as a gap and culprit-PR hunts report 'no culprit PR identified' rather than blaming an unrelated repo. - README 'Demo run' section replaced with a generic 'Run' section (no seeded build, no hardcoded repo). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…interactive) Previously the gate silently recorded 'product repo: gap' whenever corroboration failed — which kills the mandatory culprit-PR hunt without telling the human sitting at the terminal. Now, when the repo can't be corroborated against the failure signatures AND no PRs were supplied, the product repo is treated as non-assumable AND load-bearing and earns the single consolidated gate question (interactive only): 'Failures look like <domain>; which repo owns that code? (reply none → RCA without culprit-PR attribution).' Headless still records the gap and proceeds RCA-only. Preserves both invariants: never blame the wrong repo, and never silently cripple culprit-PR hunting when a human could answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…etail in Claude The end-of-run glimpse was leaking exactly what's supposed to live only on the TRA UI: per-test root causes, culprit/related PRs, cluster breakdowns, confidence rationales. renderGlimpse now emits just 'RCA analysis complete — build <id>' + a status-count line (N tests · R resolved · P pending · F failed); the caller adds the 'Full report on the Test Observability UI: <link>' line. Step 6 and the example updated to match; a test asserts no testRunId/cluster/root_cause/PR ever appears in the output. Humans open the link for the why. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ruturaj-browserstack and others added 12 commits June 23, 2026 22:26

chore(rca): gitignore local planning docs

3d22dfe

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

ruturaj-browserstack requested a review from a team as a code owner June 23, 2026 16:59

ruturaj-browserstack and others added 15 commits June 23, 2026 22:57

chore(rca): remove the k8s-rengg-tfa skill

a8c2014

Environment-specific harness removed; k8s stays a generic gate-probed capability (discoveryHints now empty, like logs/metrics). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ruturaj-browserstack force-pushed the feat/generic-rca-agent-plugin branch from 4e037b2 to a8c2014 Compare July 14, 2026 08:27

ruturaj-browserstack and others added 5 commits July 14, 2026 15:40

docs(rca): fix orphaned demo-step lines left by the k8s-skill removal…

af0f188

…; note prod default base URL Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(rca): generic multi-client RCA agent plugin#1

feat(rca): generic multi-client RCA agent plugin#1
ruturaj-browserstack wants to merge 33 commits into
mainfrom
feat/generic-rca-agent-plugin

ruturaj-browserstack commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ruturaj-browserstack commented Jun 23, 2026

What this is

Architecture

Key decisions

Testing

Install / usage

Post-Deploy Monitoring & Validation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant