feat: shard mutation testing by target instead of by test class - #40
Open
maks-oleksyuk wants to merge 1 commit into
Open
feat: shard mutation testing by target instead of by test class#40maks-oleksyuk wants to merge 1 commit into
maks-oleksyuk wants to merge 1 commit into
Conversation
This was referenced Aug 7, 2026
maks-oleksyuk
force-pushed
the
pr/mutation-shard-targets
branch
from
August 11, 2026 19:41
13ea947 to
c3368d4
Compare
Mutation runs are sharded by the files they mutate, so every mutation is generated and tested exactly once across the whole set of shards. Each shard's timeout window stays derived from the reference test suite duration recorded by the unsharded `--update-shards` run, since a shard's own initial run is shorter and would otherwise cut off mutations that a full run kills honestly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
maks-oleksyuk
force-pushed
the
pr/mutation-shard-targets
branch
from
August 11, 2026 20:24
c3368d4 to
d5c5169
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--mutate --shardsplits test classes, but a mutation belongs to a source file, andcovers()lets several test classes cover the same file. When those classes land in different shards, the file's mutations are generated in each of them, and each shard runs only its own subset of the covering tests, so a mutation killed by a test in another shard is reported as escaped.Measured on a suite of 37 test classes and 671 mutations split into 4 shards: 841 mutation runs instead of 671, 47 escaped mutations instead of 2, and an aggregate score of 94.40% against the 99.70% of an unsharded run. Per-shard
--minis not comparable to--minon a full run.What changed
The unit of sharding becomes the mutated file.
MutationRepository::units()reports, per file, the total time its mutations took and the test classes covering them, keyed by a path relative to the root so the file is portable between the machine that writes it and the one that reads it.Shard::useTimingsFile()keeps those units intests/.pest/mutation-shards.json, apart from the test timings a plain--update-shardswrites, because mutation time does not correlate with test time. A test class covering a large source file may run in milliseconds while generating hundreds of mutations.Mutation generation is narrowed to the files a shard owns inside
MutationTestRunner::run(), not through--pathor--class. Passing either of those lifts the__pest_mutate_onlygroup and makes every shard run the whole suite instead of only the tests declaringcovers().A mutation's timeout window stays derived from the reference suite duration recorded by the unsharded run. A shard's own initial run is shorter, so without this a mutation that a full run kills honestly is cut off as a timeout instead.
--log-json=<file>writes the counters of a run. Shards own disjoint sets of mutations, so adding up the file of every shard reproduces the score of a single unsharded run exactly.Result
Same suite, same 4 shards: 671 mutation runs, 2 escaped, aggregate score 99.70%, all three matching an unsharded run exactly. The critical path drops from 337s under round-robin sharding to 135s, and the spread between the slowest and fastest shard from 3.66x to 1.25x.
Usage
The first command is a periodic job that records the units; commit the file it writes so the shard jobs can read it. Adding up the
--log-jsonfiles of all shards gives the score of the whole suite.Depends on
Requires the units and
Shard::selectedUnits()added in pestphp/pest#1829, which has to be merged and released first.The
pestphp/pestconstraint here points at5.2.0, the next minor and the earliest release that can carry that API. It needs setting to the actual version at release time.Together with pestphp/pest#1829 this implements the request in pestphp/pest#1691.