Skip to content

feat: shard mutation testing by target instead of by test class - #40

Open
maks-oleksyuk wants to merge 1 commit into
pestphp:5.xfrom
maks-oleksyuk:pr/mutation-shard-targets
Open

feat: shard mutation testing by target instead of by test class#40
maks-oleksyuk wants to merge 1 commit into
pestphp:5.xfrom
maks-oleksyuk:pr/mutation-shard-targets

Conversation

@maks-oleksyuk

@maks-oleksyuk maks-oleksyuk commented Aug 7, 2026

Copy link
Copy Markdown

--mutate --shard splits test classes, but a mutation belongs to a source file, and covers() lets several test classes cover the same file. When those classes land in different shards, the file's mutations are generated in each of them, and each shard runs only its own subset of the covering tests, so a mutation killed by a test in another shard is reported as escaped.

Measured on a suite of 37 test classes and 671 mutations split into 4 shards: 841 mutation runs instead of 671, 47 escaped mutations instead of 2, and an aggregate score of 94.40% against the 99.70% of an unsharded run. Per-shard --min is not comparable to --min on a full run.

What changed

The unit of sharding becomes the mutated file. MutationRepository::units() reports, per file, the total time its mutations took and the test classes covering them, keyed by a path relative to the root so the file is portable between the machine that writes it and the one that reads it.

Shard::useTimingsFile() keeps those units in tests/.pest/mutation-shards.json, apart from the test timings a plain --update-shards writes, because mutation time does not correlate with test time. A test class covering a large source file may run in milliseconds while generating hundreds of mutations.

Mutation generation is narrowed to the files a shard owns inside MutationTestRunner::run(), not through --path or --class. Passing either of those lifts the __pest_mutate_only group and makes every shard run the whole suite instead of only the tests declaring covers().

A mutation's timeout window stays derived from the reference suite duration recorded by the unsharded run. A shard's own initial run is shorter, so without this a mutation that a full run kills honestly is cut off as a timeout instead.

--log-json=<file> writes the counters of a run. Shards own disjoint sets of mutations, so adding up the file of every shard reproduces the score of a single unsharded run exactly.

Result

Same suite, same 4 shards: 671 mutation runs, 2 escaped, aggregate score 99.70%, all three matching an unsharded run exactly. The critical path drops from 337s under round-robin sharding to 135s, and the spread between the slowest and fastest shard from 3.66x to 1.25x.

Usage

pest --mutate --covered-only --parallel --update-shards
pest --mutate --covered-only --parallel --shard=1/4 --log-json=shard-1.json

The first command is a periodic job that records the units; commit the file it writes so the shard jobs can read it. Adding up the --log-json files of all shards gives the score of the whole suite.

Depends on

Requires the units and Shard::selectedUnits() added in pestphp/pest#1829, which has to be merged and released first.

The pestphp/pest constraint here points at 5.2.0, the next minor and the earliest release that can carry that API. It needs setting to the actual version at release time.

Together with pestphp/pest#1829 this implements the request in pestphp/pest#1691.

Mutation runs are sharded by the files they mutate, so every mutation is
generated and tested exactly once across the whole set of shards.

Each shard's timeout window stays derived from the reference test suite
duration recorded by the unsharded `--update-shards` run, since a shard's
own initial run is shorter and would otherwise cut off mutations that a
full run kills honestly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@maks-oleksyuk
maks-oleksyuk force-pushed the pr/mutation-shard-targets branch from c3368d4 to d5c5169 Compare August 11, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant