Make the deterministic shell command E2E test platform-neutral

[roblourens]

Draft. Follow-up to #327489, taking the first concrete step toward running the Agent Host E2E shell tests on Windows.

The main purpose of this PR is to get a Windows CI result. The changes are believed to make one test platform-neutral, but that is an inference from reading the replay code — it has not been proven on a real Windows run. See "What to look for in CI" below.

Background

16 tests are disabled on Windows. Almost all of them trace back to a POSIX-only command string frozen into a checked-in fixture — and no test author chose those strings. The prompt described what to do, the model picked how, and whatever it picked became the fixture.

The clearest case is the test ported here. runs a deterministic shell command was skipped on Windows for all three providers, but Copilot's recorded command was already echo SHELL_VALUE_73, which runs fine on Windows. It was blocked only because Claude happened to pick printf '%s\n' "SHELL_VALUE_73".

What changed

Porting one test surfaced four separate couplings, only one of which was the command:

1. The prompt now pins the command instead of describing it. Re-recorded against live CAPI, Claude switched from printf to echo — confirming the model's choice was the whole problem. (per-test work)
2. Tool name in the AHP snapshot. The Copilot CLI names its shell tools after the shell it runs (bash on POSIX, powershell on Windows) and that name reaches the client verbatim in chat/toolCallStart. Snapshots now record ${shell}. (central)
3. Tool name in the CAPI fixture. The same name appears in the tool_use block that drives replay, so a macOS recording would instruct a Windows agent to call a tool that does not exist. Captures now store a placeholder and CapiReplayProxy expands it to the running platform's name on replay. (central)
4. Empty reasoning parts. A provider can open a reasoning part and close it without emitting a delta. Live recording sees it, but replay rebuilds the stream from the fixture's aggregated content where it leaves no trace — so any snapshot recorded during such a turn was permanently unreplayable. These are now dropped during projection. (central, and unrelated to platforms)

Also included:

• CRLF normalization in normalizeSnapshotText. Snapshot normalization handled paths, user names, shell ids and ls -l columns but not line endings, so a snapshot carrying literal tool output would fail on Windows for a reason unrelated to the behavior under test.
• shellToolReplayEnabled split into two flags. It conflated "the provider's shell-tool replay is unstable on Linux" with "this capture contains a POSIX-only command"; only the second is about Windows, and only for captures that haven't been ported. Ported tests use portableShellToolReplayEnabled.

Tests

capiReplayProxyShellTools.test.ts covers the capture/replay round-trip for shell tool names. The platform is an injectable parameter rather than a module-load read of process.platform, so both platform branches are asserted from either host — otherwise the Windows half would be dead code on a developer machine, which is exactly the branch this work exists to protect.

What to look for in CI

On the Windows leg, runs a deterministic shell command currently reports as skipped for all three providers:

- runs a deterministic shell command    (Claude)
- runs a deterministic shell command    (Codex)
- runs a deterministic shell command    (Copilot)

Success is Claude and Copilot flipping to ✔. Codex should stay skipped — it is gated on stableNewScenarioResponse: false, which is unrelated to platform.

If it fails, that is still a useful result: it means something beyond these four couplings is platform-specific, and it is much cheaper to learn that on one test than after porting all 15.

Verification so far

• Full E2E suite: 150 passing, 0 failing, stable across repeated runs (macOS)
• capiReplayProxyShellTools.test.ts: 3 passing
• typecheck-client, valid-layers-check, hygiene: pass
• One row removed from the Windows-disabled table in KNOWN_ISSUES.md

Follow-ups (not in this PR)

• Port the remaining shell-dependent tests using the checklist recorded in KNOWN_ISSUES.md
• Add a record-time lint that fails a recording containing POSIX-only constructs, so ported tests do not silently regress on the next re-record

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

[Copilot]

Review details

Comments suppressed due to low confidence (3)

src/vs/platform/agentHost/test/node/e2e/suites/fileOperationsSuite.ts:183

• The updated prompt now requires replying with the exact value only, but this assertion still passes if extra text is included (e.g. explanations or formatting). To ensure the test is actually enforcing the intended behavior (and to reduce provider variance), tighten the assertion to match the full response (after any standard trimming performed by the harness).

		assert.match(result.responseText, /SHELL_VALUE_73/);

src/vs/platform/agentHost/test/node/e2e/harness/ahpSnapshot.ts:472

• This mapping duplicates the shell-tool placeholder logic introduced in capiReplayProxy.ts (SHELL_TOOL_PREFIXES / SHELL_TOOL_PLACEHOLDERS). Keeping two independent maps increases the risk of divergence (e.g. adding a new prefix/tool variant in one place but not the other). Consider centralizing the mapping in a shared helper (or exporting it from one module) so both AHP snapshot normalization and CAPI fixture normalization stay consistent.

function normalizeShellToolName(toolName: string): string {
	const shellToolPlaceholders: Record<string, string> = {
		bash: '${shell}', powershell: '${shell}',
		read_bash: '${read_shell}', read_powershell: '${read_shell}',
		write_bash: '${write_shell}', write_powershell: '${write_shell}',
		stop_bash: '${stop_shell}', stop_powershell: '${stop_shell}',
		bash_shutdown: '${shell_shutdown}', powershell_shutdown: '${shell_shutdown}',
		list_bash: '${list_shell}', list_powershell: '${list_shell}',
	};
	return shellToolPlaceholders[toolName] ?? toolName;
}

src/vs/platform/agentHost/test/node/e2e/harness/ahpSnapshot.ts:446

• The new projection behavior is central to making previously-recorded snapshots replayable (dropping empty reasoning parts). Adding a focused unit test for dropEmptyReasoningParts (or for snapshot serialization with an empty reasoning part present) would protect against regressions and ensure the intended unreplayable-case is covered.

function dropEmptyReasoningParts(actions: object[]): object[] {
	return actions.filter(entry => {
		const action = (entry as { action?: { type?: string; part?: { kind?: string; content?: string } } }).action;
		return !(action?.type === ActionType.ChatResponsePart
			&& action.part?.kind === ResponsePartKind.Reasoning
			&& action.part.content === '');
	});
}

• Files reviewed: 18/18 changed files
• Comments generated: 1
• Review effort level: Low