optimizer: add prefix-unique join order scoring behind a flag

[frankmcsherry]

Motivation

Join ordering scores each candidate order extension only by target-side uniqueness: unique_key says the incoming relation is unique on its bound columns, so the join cannot grow the carried prefix. The dual signal is missing. When the placed prefix is unique on the equated expressions, each incoming row matches at most one prefix row, so the extended result is bounded by the incoming relation's size. Foreign-key equivalences traversed backwards (unique side placed, fk side incoming) are exactly this shape, the bound composes down FK chains, and delta join paths starting from the "many" end of a chain cannot avoid such edges. Today they score as unbounded mysteries: in a chain a.b_fk = b.b_pk AND b.c_fk = c.c_pk with everything indexed, the path from c reads %2:c » %1:b[c_fk]KA » %0:a[b_fk]KA, indistinguishable from a genuine many-to-many hazard.

Change

The Orderer now tracks unique keys of the placed prefix, folding in each placement with the standard join key rule: forward-unique keeps the old keys, reverse-unique adopts the incoming relation's keys, and otherwise pairwise unions (which is what lets the bound compose along chains). Candidates gain a prefix_unique characteristic ranked immediately below unique_key, in a new JoinInputCharacteristicsV3 following the V1/V2 versioning precedent. V3 is selected by a new default-off, cluster-scoped flag enable_join_reverse_edge_scoring, and builds on V2 (enable_join_prioritize_arranged) ordering. EXPLAIN prints the new field as P and accepts enable join reverse edge scoring as a WITH option.

A cute corollary: a cross join against a prefix with the empty unique key (at most one row, e.g. a scalar aggregate) counts as bounded, so singleton cross joins stop being feared.

Known conservatism, documented in the code: characteristics are computed at push time, so a queued candidate does not benefit from prefix keys established after its last push. Sound (never overclaims), just conservative.

Verification

• With the flag off, plans are unchanged: 2349/2349 sqllogictest queries pass unmodified across transform/, explain/, joins.slt, chbench.slt, and both TPCH files, and rewriting the new corpus reproduces every baseline byte-identical.
• New test: test/sqllogictest/transform/join_reverse_edges.slt, FK chain / star / snowflake shapes with captured plans flag-off and flag-on.
• Unit tests for mz-expr, mz-sql-parser, and mz-sql (including the optimizer-features vars invariant) pass.

Evidence from existing corpora (flag on)

Three-way attribution over 110 EXPLAIN queries in chbench, TPCH (index + MV), joins, and ldbc_bi, comparing V1 (production default), V2, and V3, with the P marker stripped before comparison so cosmetic changes don't count: 22 identical, 36 marker-only, 4 changes inherited from V2's arranged-first ranking, 44 reranked by the new signal (plus 4 mixed). Sampled judgments: binary sub-joins now start from the bounded side (no cost, both sides of a differential join are arranged anyway). The tradeoff deserving review: on delta paths, P above A means a bounded-unarranged key can beat an arranged-unbounded key, buying a cardinality bound at the price of an arrangement, the same philosophy as unique_key > arranged in V2 extended one level.

Open questions

• Should V3 extend V1 instead of V2, keeping this flag independent of enable_join_prioritize_arranged?
• Is prefix_unique > arranged the right rank, given the arrangement-vs-bound tradeoff above?
• CREATE CLUSTER ... FEATURES is deliberately not wired for the new flag (EXPLAIN WITH and the system var are). Should it match enable_join_prioritize_arranged?

🤖 Generated with Claude Code