Speed up the test suite

.github/workflows/ci.yml

@@ -20,6 +20,20 @@ jobs:
         with:
           node-version: 24
           cache: pnpm
+      # Cache keys end in github.sha so every run saves a fresh cache; restore-keys
+      # fall back to the newest cache whose lockfile/config hash still matches.
+      - uses: actions/cache@v4
+        with:
+          path: .turbo/cache
+          key: turbo-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'turbo.json', 'vitest.shared.ts') }}-${{ github.sha }}
+          restore-keys: |
+            turbo-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml', 'turbo.json', 'vitest.shared.ts') }}-
+            turbo-${{ runner.os }}-
+      - uses: actions/cache@v4
+        with:
+          path: ~/.cache/silk-effect/native
+          key: silk-native-${{ runner.os }}-${{ github.sha }}
+          restore-keys: silk-native-${{ runner.os }}-
       - run: pnpm install --frozen-lockfile
       - run: pnpm check
       - run: pnpm release:candidate

AGENTS.md

@@ -206,6 +206,33 @@ Do not build a `ManagedRuntime` test harness, call `Effect.runPromise` or `Effec
 each test, rebuild common layers per test, or wrap Effect code in `async` callbacks. A test that
 genuinely needs isolation may scope a distinct layer within that test.
 
+## Keep tests cheap
+
+The compiler suite is the critical path of `pnpm check`; every test pays for the compiler
+pipelines it runs. Prove each claim at the cheapest tier that can falsify it.
+
+- Prove language semantics with `Analysis.evaluate`. Add a wasm leg only when the claim is about
+  wasm codegen. Add a native leg only when lowering is genuinely target-specific: syscalls,
+  suspension frames, drop hooks, recursion stack bounds, or native allocation metrics.
+- Never add a per-feature "the native binary agrees" test. That claim is proven differentially by
+  `DriverNativeAcceptance` — add your program to `test/support/corpus.ts` instead of calling
+  `Driver.compile` in a feature file.
+- Do not write per-feature fresh-process determinism tests. Fresh-process determinism is a global
+  property, guarded by the designated canary determinism tests; per-feature determinism is proven
+  by committed-golden byte comparisons in-process.
+- Build one `Analysis` snapshot per source program per file and share it across assertions and
+  engines. Do not re-run `ofSourceRealized` on the same source.
+- Assert diagnostic codes and spans, not message text. The generated diagnostic catalog gates
+  wording.
+- No timing assertions, byte counts, or instruction counts in the correctness suite. Structural
+  claims assert structure; performance claims live in opt-in bench targets.
+- In failure-ordinal and stress sweeps, run the evaluator and wasm at every point; run native only
+  at boundary points (first failure, one mid-growth, completion).
+- Prefer adding a case to an existing file over creating a new test file: each new file costs
+  ~0.5s of worker startup and re-imports the compiler.
+- A test that cannot fail for a reason distinct from its neighbors is not a test; delete it rather
+  than keeping it for coverage optics.
+
 ## Scope resource lifecycles
 
 Use `Effect.acquireRelease`, `Effect.acquireUseRelease`, or an equivalent scoped bracket whenever

openspec/changes/archive/2026-08-15-speed-up-test-suite/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-15

openspec/changes/archive/2026-08-15-speed-up-test-suite/baseline.md

@@ -0,0 +1,71 @@
+# Baseline: packages/compiler test durations
+
+Run 2026-08-15 on merge of origin/main (691092e), Apple Silicon, all cores.
+Wall 184.0s; sum of per-file time 1673s; 233 files, 1852 tests.
+Known failure (pre-existing on main): WasmShadowStackHeapCollision host-stack control.
+
+| # | File | s | tests |
+|---|------|---|-------|
+| 1 | LexerPressure.test.ts | 143.3 | 6 |
+| 2 | TemporaryDirectoryAcceptance.test.ts | 130.4 | 6 |
+| 3 | StackVmPressure.test.ts | 82.3 | 5 |
+| 4 | VectorAcceptance.test.ts | 62.9 | 20 |
+| 5 | UnicodeNormalization.test.ts | 50.5 | 8 |
+| 6 | DriverNativeAcceptance.test.ts | 48.5 | 1 |
+| 7 | EffectSuspensionNative.test.ts | 45.7 | 5 |
+| 8 | SynchronousEffectCost.test.ts | 41.3 | 1 |
+| 9 | StoredCallableRuntime.test.ts | 38.7 | 5 |
+| 10 | StoredCallableDeterminism.test.ts | 38.6 | 2 |
+| 11 | OsFileSystem.test.ts | 37.9 | 10 |
+| 12 | Driver.test.ts | 32.4 | 15 |
+| 13 | EffectSuspensionComposition.test.ts | 31.3 | 11 |
+| 14 | HashedCollectionDeterminism.test.ts | 27.7 | 4 |
+| 15 | ChildProcess.test.ts | 26.2 | 13 |
+| 16 | StoredEffectEngineParity.test.ts | 25.9 | 4 |
+| 17 | StackVmPressureDeterminism.test.ts | 25.5 | 1 |
+| 18 | HashedCollectionOwnership.test.ts | 25.2 | 4 |
+| 19 | LexerPressureDeterminism.test.ts | 25.0 | 1 |
+| 20 | WasmBackend.test.ts | 25.0 | 50 |
+| 21 | IntegerScalars.test.ts | 24.9 | 6 |
+| 22 | AlgorithmExamples.test.ts | 23.6 | 4 |
+| 23 | HashedCollections.test.ts | 23.3 | 7 |
+| 24 | HostInput.test.ts | 23.2 | 10 |
+| 25 | SlotLaneWidth.test.ts | 23.2 | 14 |
+| 26 | UnicodeNormalizationConformance.test.ts | 23.2 | 2 |
+| 27 | RecursionStackBoundary.test.ts | 18.5 | 10 |
+| 28 | VectorSort.test.ts | 17.5 | 13 |
+| 29 | BootstrapEvaluation.test.ts | 17.3 | 29 |
+| 30 | EffectRuntime.test.ts | 17.2 | 18 |
+| 31 | MultiAffineReturn.test.ts | 16.7 | 2 |
+| 32 | ScannerAcceptance.test.ts | 16.6 | 2 |
+| 33 | NumberText.test.ts | 15.9 | 6 |
+| 34 | BulkMemory.test.ts | 15.5 | 7 |
+| 35 | UserServices.test.ts | 14.7 | 11 |
+| 36 | ModuleVerification.test.ts | 14.4 | 1 |
+| 37 | OwnedAllocationDispatch.test.ts | 12.9 | 4 |
+| 38 | WasmShadowStackHeapCollision.test.ts | 12.8 | 5 |
+| 39 | FloatMath.test.ts | 12.4 | 10 |
+| 40 | Logging.test.ts | 12.0 | 7 |
+| 41 | FileSystemAcceptance.test.ts | 10.7 | 8 |
+| 42 | EffectSuspensionEvaluation.test.ts | 10.7 | 5 |
+| 43 | ResultStdlib.test.ts | 10.2 | 10 |
+| 44 | IntrinsicCatalog.test.ts | 10.0 | 6 |
+| 45 | BoxHeapIndirection.test.ts | 8.6 | 10 |
+| 46 | StringAcceptance.test.ts | 8.5 | 4 |
+| 47 | IfThenElseAcceptance.test.ts | 7.9 | 10 |
+| 48 | OwnedAllocation.test.ts | 7.6 | 10 |
+| 49 | ScannerDeterminism.test.ts | 7.5 | 1 |
+| 50 | ZipAcceptance.test.ts | 7.3 | 10 |
+| 51 | DropHookExecution.test.ts | 6.9 | 3 |
+| 52 | LlvmIrRoundTrip.test.ts | 6.5 | 4 |
+| 53 | EditorIntelligence.test.ts | 6.3 | 25 |
+| 54 | BoundOperationWitness.test.ts | 6.2 | 18 |
+| 55 | StaticByteViewIndexing.test.ts | 6.1 | 4 |
+| 56 | LoggingDeterminism.test.ts | 5.8 | 1 |
+| 57 | AllocationMetricsAcceptance.test.ts | 5.6 | 4 |
+| 58 | Suspendability.test.ts | 5.4 | 8 |
+| 59 | Elaboration.test.ts | 5.1 | 82 |
+| 60 | StoredCallableDiagnostic.test.ts | 5.0 | 11 |
+
+Top 10 files: 682s (41% of per-file total). Top 30: 1176s (70%).
+Determinism family: 23 files, 153s.

openspec/changes/archive/2026-08-15-speed-up-test-suite/design.md

@@ -0,0 +1,64 @@
+# Design
+
+## Context
+
+See proposal.md — Why. Measured facts this design relies on (six spikes, 2026-08-15):
+
+- One native `Driver.compile` ≈ 1.05s = 64% JS LLVM-bitcode backend + 24% frontend + 11% clang + <1% binary execution. Native tests are slow because of the extra JS pipeline, not the toolchain.
+- Clang-touching test files: 66 of 225, ~70% of per-file wall. Fresh-process determinism files: 24, ~209s. Vitest harness: ~4–5% of suite CPU. Slowest files: `LexerPressure` 226s (162s in one failure-ordinal sweep), `TemporaryDirectoryAcceptance` 193s, `StackVmPressure` 115s.
+- `DriverNativeAcceptance.test.ts` already differential-tests interpreter vs native over a ~70-program corpus (`test/support/corpus.ts`) at ~0.77s/program, serially.
+- `NativeToolchain.makeDiskArtifactCache` exists and works (key = sha256 of kind/triple/profile/clang/shim/bitcode; measured hit: 1.76s → 4ms for the toolchain step); `defaultArtifactCache()` is a process-local `Map`; `SILK_NATIVE_CACHE_DIR` is set by `packages/compiler/vitest.config.ts` and read by nothing.
+- Ruled out by measurement: bun (2x slower on compiler JS, incompatible with `@effect/vitest`), plain-script conversion (~4–5% ceiling, loses timeouts/isolation), `--no-isolate` (no wall change), clang `-O0` (no-op), clang batching (clang is ~2%).
+
+## Goals / Non-Goals
+
+**Goals**
+- Cut compiler-suite CPU ~35–50% by deleting redundant tests and tiering expensive legs, without losing any distinct failure mode the suite can currently catch.
+- Make determinism and native-agreement coverage *centralized* (canaries + corpus) so the marginal cost of a new language feature's tests is one corpus entry, not a new clang-spawning file.
+- Stop regrowth: encode the cost rules where AI agents read them (AGENTS.md).
+- Warm caches everywhere a run can be warm (CI, worktrees, cross-process native artifacts).
+
+**Non-Goals**
+- No `Analysis` snapshot sharing or stdlib elaboration memoization (compiler-side; follow-up change).
+- No harness replacement (vitest stays), no worker/pool retuning beyond what exists.
+- No behavior change to the compiler beyond the opt-in disk cache default.
+
+## Decisions
+
+**D1. Determinism: 3 canaries + in-process goldens, not 23 fresh-process files.**
+Fresh-process determinism catches nondeterminism whose source is process-local state (map iteration order, hashing seeds, pointer-derived ordering). That class is global to the compiler, not per-feature: any sufficiently rich program that exercises the full artifact surface will surface it. Keep `ScannerDeterminism` (stdlib imports, allocation, both release backends, HIR/ownership/MIR encodings, 7.5s), `ConditionalConformanceDeterminism` (generics, conformance memo order, both backends, 1.0s), and `StoredCallableDeterminism` (callable environments, native+wasm execution, 38.6s). `LlvmWasmDeterminism` (baseline pick) measured as a trivial-identity program whose whole surface the other canaries subsume — it is deleted with the rest. Every deleted file's per-feature byte-identity claim remains enforced by its committed-golden comparisons, which run in-process. Alternative considered: keep all 23 but share one spawned process per file — rejected, still pays 2 full release pipelines × 20 files for no added failure mode.
+
+**D2. Native agreement: corpus-only, with an explicit target-specific allowlist.**
+`DriverNativeAcceptance` is already the designated differential gate. Feature files' programs move into `test/support/corpus.ts`; their standalone `Driver.compile` + `spawnSync` legs are deleted. Native legs stay *only* where lowering is genuinely target-specific and the corpus's exit-code differential cannot express the claim: `EffectSuspensionNative`, `DropHookExecution`, `RecursionStackBoundary`, syscall-touching tests (`OsFileSystem`, `TemporaryDirectoryAcceptance`, `HostInput`, `StandardStreams`), and allocation-metrics tests. The allowlist is written down in AGENTS.md so the burden of proof is on adding a native leg, not removing one.
+
+**D3. Failure-ordinal sweeps: evaluator+wasm carry every ordinal, native carries boundaries.**
+The sweep's claim (typed `OutOfMemory`, exactly-once release, no partial exposure) is a semantics claim the evaluator and wasm engines check cheaply per ordinal. Native's unique contribution is leak evidence through the real allocator — preserved at first-failure, one mid-growth, and completion ordinals. This converts O(ordinals) native pipelines into O(3) per pressure program and removes the two worst single tests in the suite. The quota constant currently embedded in source per iteration also defeats the artifact cache; boundary-only native legs make that moot.
+
+**D4. Disk cache: flip the default, don't touch 66 call sites.**
+`defaultArtifactCache()` in `packages/compiler/src/NativeToolchain.ts` returns `makeDiskArtifactCache(process.env.SILK_NATIVE_CACHE_DIR)` when the variable is set, else the existing Map. Every existing `Driver.compile` caller inherits it; the vitest config env line becomes live as originally intended. Two known limitations, accepted and documented: the key hashes the clang *path* (stale after a clang upgrade at the same path — mitigated by including clang version in the key while we're there), and no eviction (mitigated: CI cache is bounded by the actions/cache limit; local dir is small — 22MB after weeks of the old spike). Honest sizing: a hit skips only the clang step (~11% of a compile) plus the shim compile, so this is a small steady win, not the headline.
+
+**D5. CI/worktree caches.**
+`actions/cache` on `.turbo` keyed by lockfile+turbo config hash with restore-keys fallback, and on `SILK_NATIVE_CACHE_DIR`. Worktrees: `scripts/turbo.mjs` sets `TURBO_CACHE_DIR` to a repo-adjacent shared path when the checkout is under `.claude/worktrees/` (turbo hashes are repo-relative, so cross-worktree hits are sound).
+
+**D6. Perf assertions leave the correctness suite.**
+`SynchronousEffectCost` keeps only its *structural* assertions (entry structure omits foldable constructor calls — spec-mandated) and drops exact byte/branch counts and timing rounds; `OccurrencePerformance` is deleted (timed rounds on shared CI are a flake generator, and the spec makes no performance claim).
+
+**D7. Verification is measured, not asserted.**
+Before the deletion PR: one `vitest run --reporter=verbose` baseline, committed to the change as a ranking. After each phase: same run, diff the totals. Deletions must also pass a mutation-style spot check: for 3 sampled deleted files, re-introduce a representative historical bug (or revert its fixing commit locally) and confirm a surviving test still fails.
+
+## Risks / Trade-offs
+
+- [Deleting a test that was the only guard for a real regression] → D7's spot check; deletions grouped by pattern in separate commits so `git revert` restores a whole family; corpus entries land in the same commit as the leg they replace.
+- [Canary set misses a feature-specific nondeterminism source] → canaries chosen to cover the full artifact surface (both backends, stdlib, generics, callables); goldens still catch any in-process nondeterminism per feature; a future nondeterminism escape adds a canary, not 20 files.
+- [Boundary-ordinal native sweep misses a native-only leak at an interior ordinal] → interior ordinals still checked by wasm (linear memory) and evaluator (logical releases); native allocator behavior does not vary by ordinal index, only by rollback path shape, and the boundary set covers all three path shapes (immediate failure, partial init, full success).
+- [Disk cache serves a stale artifact after toolchain change] → add clang version to the key in the same change; corrupted/missing entries recompile per the spec delta.
+- [Turbo cache sharing across worktrees races concurrent runs] → turbo's cache writes are atomic (content-addressed files); worst case is a redundant write.
+
+## Migration Plan
+
+Four independent PRs in order of value: (1) deletions + corpus folds + AGENTS.md rules, (2) pressure-loop tiering, (3) cache wiring (toolchain default + CI + worktrees), (4) bench extraction. Each is revertible alone; specs archive after all four land.
+
+## Open Questions
+
+- Which stored-callable determinism file becomes the third canary (pick the one with widest artifact surface when implementing — likely `StoredCallableDeterminism` if it covers environments + generics).
+- Whether `EditorIntelligence.test.ts` moves to the new IDE package before or after its duplicate cases are pruned (sequencing only; either order works).

openspec/changes/archive/2026-08-15-speed-up-test-suite/proposal.md

@@ -0,0 +1,36 @@
+# Speed up the test suite
+
+## Why
+
+`pnpm check` takes ~20 minutes and keeps growing. Six measurement spikes (2026-08-15) located the cost: `packages/compiler` is ~2,580s of the workspace's ~3,100s test CPU, and the dominant waste is redundant full compiler pipelines — 23 per-feature fresh-process determinism files, ~100 per-feature native legs already subsumed by the `DriverNativeAcceptance` differential corpus, and pressure-test rollback loops that run a full native compile per quota ordinal. The native toolchain itself is only ~11% of a native test's cost; vitest overhead is ~4–5%; bun and plain-script conversions were measured and ruled out.
+
+## What Changes
+
+- **Delete redundant compiler tests** (~25% of cases, est. 700–1,200s CPU): keep 3 fresh-process determinism canaries and delete the other 20 `*Determinism.test.ts` files (+ fixtures); fold per-feature "native binary agrees" programs into `test/support/corpus.ts` and drop their standalone `Driver.compile` legs; remove engine-matrix duplicate files, exact diagnostic-message-string assertions (the generated catalog gates wording), and duplicated `EditorIntelligence` cases.
+- **Move performance measurements out of the correctness suite**: `SynchronousEffectCost` and `OccurrencePerformance` become an opt-in bench target (or are deleted).
+- **Trim pressure-loop native legs** (~200–280s): `LexerPressure` and `StackVmPressure` failure-ordinal sweeps run natively only at boundary ordinals; evaluator and wasm carry intermediate ordinals.
+- **Wire the dead native artifact cache**: `NativeToolchain.defaultArtifactCache()` returns a disk cache when `SILK_NATIVE_CACHE_DIR` is set (the env var is currently set by `packages/compiler/vitest.config.ts` and consumed by nothing).
+- **Persist caches in CI and across worktrees**: `actions/cache` for `.turbo` (and the native cache dir) in CI; shared `TURBO_CACHE_DIR` for `.claude/worktrees/*`.
+- **Add a "Keep tests cheap" section to AGENTS.md** so AI-written tests stop regrowing the waste: cheapest-tier proof obligation, corpus-first native coverage, no per-feature determinism tests, no timing assertions, snapshot reuse.
+
+Not in scope (follow-up candidates): `Analysis` snapshot sharing across engines and stdlib elaboration memoization (~150–250s+, compiler-side work that also benefits the LSP).
+
+## Capabilities
+
+### New Capabilities
+
+_None._
+
+### Modified Capabilities
+
+- `bootstrap-compiler-driver`: fresh-process determinism is consolidated from per-feature gates into designated canary gates; per-feature engine-agreement obligations are discharged by the aggregate differential corpus rather than standalone per-feature native tests.
+- `bootstrap-language-pressure-programs`: failure-ordinal sweeps are carried by the evaluator and WebAssembly engines, with native execution required only at boundary ordinals; cross-engine agreement remains required for representative acceptance cases.
+- `bootstrap-native-toolchain`: the default artifact cache honors `SILK_NATIVE_CACHE_DIR`, persisting compiled artifacts on disk keyed by content so identical requests skip clang across processes and runs.
+
+## Impact
+
+- `packages/compiler/test/**` (deletions, corpus additions, pressure-loop edits), `packages/compiler/test/support/corpus.ts`
+- `packages/compiler/src/NativeToolchain.ts` (`defaultArtifactCache`), `packages/compiler/vitest.config.ts` (comment correction: key includes clang path, not version)
+- `.github/workflows/ci.yml` (turbo + native cache persistence), `scripts/turbo.mjs` (worktree-shared `TURBO_CACHE_DIR`)
+- `AGENTS.md` (new test-cost rules)
+- Expected effect: compiler-suite CPU down ~35–50% from deletions/trims alone; CI additionally gains warm turbo and native caches. No language, compiler, or public API behavior changes other than the opt-in disk cache default.