gh-153044: Auto-possessify greedy repeats with a disjoint continuation

[serhiy-storchaka]

Add an optimizer pass that turns a greedy repeat into a possessive one when this cannot change what the pattern matches: if the repeated atom and every character that could possibly follow the repeat are provably disjoint, backtracking into the repeat is always futile, so a+b is compiled as if it were a++b. PCRE2 performs the same optimization under the name "auto-possessification".

The analysis only ever fires when it can prove disjointness, so any imprecision can merely cost an optimization, never change a match result. The repeated body may be a single one-character atom or a rigid concatenation (only its leading atom matters); the follower analysis walks through group boundaries, alternations, nullable repeats, atomic groups and the \z/$ anchors. Disjointness is proved by an atom model for \d/\w/\s, by finite witness sets (case-folded under IGNORECASE), or by probing the engine's own category predicate through a new helper _sre.category_matches(), so every engine category is decided exactly: \p{Lu}+x possessifies, and a category and its complement are always disjoint, so \p{X}+\P{X} possessifies for every X. A fused set-operation charset is evaluated exactly, so [\w--\d]+\d and [a-z--b]+b possessify too.

The main benefit is failing or backtracking-heavy matches whose repeat is followed by a character set or category, which the existing REPEAT_ONE literal fast path does not cover: measured 2.2–4.5× on such failing matches, and some catastrophic-backtracking patterns are defused as a side effect. Successful matches are unaffected (~1×). Correction: where disjointness is provable, every give-back already fails at the first following atom, so this removes only a constant factor; the catastrophic nested-repeat patterns like (a+)+b are among the possible extensions.

Validated by differential fuzzing against the unoptimized compilation (millions of checks accumulated across pattern families: random ASCII, IGNORECASE with non-ASCII fold groups, scoped flags, anchors, set operations, \p{...} categories, exhaustive small-alphabet brute force), plus full-range exactness checks of the category probes against the engine.

This is the first step described in the issue; alternation bodies with disjoint first characters, word-boundary and lookahead followers, and category-vs-category reasoning are possible extensions.

• Issue: Auto-possessify greedy repeats in regular expressions #153044

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #33444493 | 📁 Comparing 7868451 against main (639a552)

  🔍 Preview build  

2 files changed

± whatsnew/3.16.html
± whatsnew/changelog.html