Multimode batched evaluation of factorized CC (cost model + placement + dry-run) - #583
Open
evaleev wants to merge 87 commits into
Open
Multimode batched evaluation of factorized CC (cost model + placement + dry-run)#583evaleev wants to merge 87 commits into
evaleev wants to merge 87 commits into
Conversation
CostProfile (peak/flops/exec) over the factorized IR via a zero-data dry-run evaluation, driving the batched cost model's predictions.
Perf-first (DenseTimeSpace) objective with peak_threshold as a ceiling; role-split (contracted/external) batchability; order-aware placement over the combined nest; per-node batch annotations consumed by the evaluator.
External-mode scatter + contracted accumulate; cache scope chain with fall-through; slice-on-use; per-level placement driven by a per-canonical lifetime mask (cross-occurrence meet) unioned with contracted residency; iterative (stack-safe) tree traversal.
Index-space occupancy predicates robust to non-physical spaces; logger; is_valid accepts Power; convention.
The multimode-batched-eval tests had only run under Release/IGNORE, so several Debug-only asserts (and one ASan bug) were masked. Fix them so the suite is green under Debug (SEQUANT_ASSERT_BEHAVIOR=ABORT + AddressSanitizer): - Covariant tensor forms in the synthetic batched-eval tests: contracted indices had been placed in the same bra/ket slot, tripping create_graph's strict-braket invariant. Reorient contractions Einstein-properly and move Hadamard/external indices to the aux slot (test_lifetime_mask, test_eval_ta, test_eval_dryrun). Physical-tensor tests were unaffected (Symm braket). - test_eval_dryrun: a rank-4 CSV composite used a duplicate proto index; use a distinct fourth occ index. - test_eval_ta: rand_tensor_yield now sizes the m (mu~) space; and copy the compared tiles by value in shape_spike_ToT_inner_contraction_to_flat_T (it bound a reference into a temporary Future -> ASan stack-use-after-scope). - test_cache_manager: the batch-axis veto is phase-2 -- a node carrying a batch mode free on its own result is batch-variant (External or Contracted) and correctly refused run-scope caching; update the stale case-3 expectation. - cost_model: seeded_root_peak_batched must admit the seed via the external-role predicate too. The seed is an external mode, so build_context's role filter gates it through is_batchable_external_index; overriding only the contracted-role predicate dropped the seed and tripped the k_seed assert. - Hide ([.]) the all-C60-terms perf-first cost diagnostic: it runs optimize() on every summand (tens of minutes in Debug) and has no correctness checks. - Add a hidden ([.]) water-20 occ-batching overcompute dry-run diagnostic.
order_aware_recompute=false selects the set-keyed DP that ignores the per-batch-block replay recompute, which under-costs every batched schedule and is never the more realistic default. Default it to true in BatchPolicy and CostParams (MPQC and other callers inherit it via optimize()). Pin it false in the two cases that specifically characterize the legacy set-keyed behavior: reconstruct_batched_modes_emits_external_per_node (its own comment documents the order-aware-off emit_external regime) and the C60 objective-determines-factorization case (peak-first forms the fully-sliceable 4-PAO only under the set-keyed peak model; under the realistic resident-scan model the contrast collapses -- peak-first also avoids it and perf-first flops then exceed peak-first because the recompute is charged).
OptimizeOptions::inner_pow and PeakBatchedModel::inner_pow already have no
default (empty + composite indices -> inner_aware_volume throws, so the old
silent mis-sizing fallback cannot recur). But 9 OptimizeOptions{...} designated
initializers omitted inner_pow, which g++ -Wextra -Werror flags as
missing-field-initializers (clang does not, so macOS CI and local clang builds
missed it) -- breaking every Linux Debug job.
Add an explicit .inner_pow = {} (composite-free no-op) at all 9 sites
(optimize.cpp compatibility_opts + 8 in test_optimize). Also finish the
removal that had missed CostParams::inner_pow: drop its stale = {} default and
its stale "sized by idxsz (k=1)" fallback comment so all three inner_pow fields
are uniformly no-default (all 21 CostParams{...} sites already set it).
The static per-node walk in cost_profile() prices each node once, so its
flops/exec are order- and batching-blind and never reflect the per-occ-block
REPLAY recompute the batched evaluator does at runtime -- the reason a dry-run
could not predict occ-batching being slower than aux-only.
Split the reported cost:
- Rename CostProfile::{flops,exec_cost,n_ops} -> model_{flops,exec,n_ops} (the
static DP-model quantities, unchanged).
- Add dryrun_{flops,exec,n_ops}, tallied from the existing Trace::On replay: an
optional CostSink is attached to the shared dry-run CostModel, and every
actual product-op execution (DryRunOps::prod) folds its own SLICED-extent
flops/exec (the same numbers already computed for the per-op OpCost log) into
it. Because a sliced occ-dependent op run N times does ~1/N work per pass, its
sliced-cost sum is work-neutral; only occ-INDEPENDENT work re-executed at full
size once per block inflates -- so dryrun_* isolates exactly the recompute.
The sink is opt-in (nullptr default) and lives on the dry-run CostModel, which
is constructed only inside cost_profile(); eval.hpp / make_evaluator are
untouched, so mpqc's production evaluator path is byte-identical.
water-20 [.][dryrun-water20-overcompute], heavy occ batching: model_flops
ratio 1.0 (flat), dryrun_flops/exec ratio ~1.98, dryrun_n_ops ~55x -- the
recompute is now visible where the model walk saw nothing.
Extend [.][dryrun-water20-overcompute] to three configs -- aux-only, occ+aux
order_aware=false (MPQC production / root-level forest seed), occ+aux
order_aware=true (node-level placement) -- and report the recompute-aware
dryrun_{exec,n_ops} per config plus the OA-true-vs-false ratio.
Diagnoses the water-20 order_aware=true slowdown: under occ batching,
order_aware=true does ~2x the dryrun_exec and ~12x the op-executions of
order_aware=false, and (via its resident-scan peak model reporting higher
peaks) also tips more terms over peak_threshold so occ batching engages where
order_aware=false leaves them un-batched -- a double hit that matches the
observed runtime regression.
Flipping the default to true (previous commit on this branch) turned on the order-aware cost model AND, together with batch_spectator_indices, node-level external placement. On water-20 that path is a runtime regression -- the water-20 dryrun diagnostic measures ~2x the traffic and ~12x the op-executions of the root-level forest seed for the same term -- and, worse, was reported on Owl (job 649250) to produce a WRONG schedule: incorrect PNO-CCSD iteration energies and malformed eval ops. Restore the known-good default (false = legacy set-keyed DP + root-level forest seed) until node-level placement is root-caused and fixed. The two tests that pin order_aware off explicitly keep passing (the pin now merely matches the default).
order_aware_recompute conflated two orthogonal concerns: the order-aware recompute COST MODEL (which factorization the DP selects) and the node-level external-mode EMISSION placement (per-node External stamps vs the root-level forest seed). node_level_placement was defined as order_aware_recompute && batch_spectator_indices, so enabling the more-realistic cost model forced the node-level emission along with it. A water-8 A/B (holding the cost model fixed, toggling only emission) shows the regression is the EMISSION, not the cost model: node-level placement runs ~6x slower (~124 vs ~20 s/iter) and emits ~8x more batch scopes than the root-level seed, because it nests a batch scope at every carrying node and the batched evaluator replays each. The order-aware cost model with root-seed emission is correct and cheap. Node-level placement also produces a wrong residual on water-20 (size-dependent; not reproduced at water-8/he10). Split node_level_placement into its own BatchPolicy/CostParams/CostModel flag (default false), threaded alongside order_aware_recompute. order_aware_recompute now drives only selection; node_level_placement drives only emission. Both default false, so this is behavior-neutral. A gated SEQUANT_NODE_LEVEL_PLACEMENT env knob forces the placement for A/B diagnostics without recompiling a flag through the caller. Tests that engaged node-level placement via order_aware_recompute=true now set node_level_placement=true explicitly; the node-level correctness sweep now drives the emission via its sweep variable.
Now that node-level emission is separately gated by node_level_placement (default off), the order-aware recompute cost model is selection-only and safe to default on: it charges recompute realistically and picks better-batching factorizations, while emission stays the correct, cheap root-level forest seed. Verified no churn across the [optimize] suite (628 assertions) and the batched emission tests. The internal CostModel/oracle-helper member defaults stay false (documented seeded-probe reason); the public BatchPolicy/CostParams path threads the true default through.
…le recompute
Add a batched-evaluation schedule-visualizer pipeline and make avoidable
recompute a first-class per-node output of cost_profile().
- schedule_dump.hpp (new): per-term IR schedule-record emitter
(schedule_ir_json) and a shared cost_op_signature() join key (result index
labels + the sorted operand pair). One definition, used by three producers
so a DAG node, its runtime Build event, and cost_profile's per-node number
all carry the identical key -- no signature is reconstructed downstream.
- eval.hpp: runtime schedule-dump hooks (SCHEDULE_RUN_EVENT / RUN_GROUP),
each internal Build event stamped with the same signature and per-loop
dependent-mode flags, all gated by SEQUANT_SCHED_DUMP (production path
byte-identical when unset).
- CostProfile gains per-node avoidable_nodes {label,count,exec,flops} plus
avoidable_exec / avoidable_ops and avoidable_time(). DryRunOps::prod tallies
each build's necessary = product of block counts of its touched (dependent)
modes, so builds - necessary is the avoidable recompute (a node rebuilt once
per block of a mode it does not touch). The cost model is dense, so necessary
is exact and no empirical correction is needed. avoidable_nodes_from_sink()
is the shared rollup, reused by the schedule-dump test to emit these numbers.
- Consolidate the two dryrun avoidable witnesses (occ-veto, extmode) onto
cost_profile's structural per-node rollup, replacing the BatchGroup/BatchIter
trace-parse string-match reconstruction with a read of cp.avoidable_* (the
structural signature is relabeling-proof; the trace is still parsed only for
the scatter/group Begin markers cost_profile does not expose).
…uckets The per-node avoidable-recompute rollup keyed only by the label signature (result+operand indices), so a node built at many slice sizes landed in one bucket. Because the exec model is roofline-like (nonlinear in slice size), those builds span orders of magnitude (a 257x spread on the C60 external-occ arm); pricing avoidable as count * a single per-build exec then exceeded the whole replay's exec -- the impossible avoidable_time > 100%. Fix: key the sink by the EXTENDED signature (label + the touched modes' realized extents), so every build in a bucket ran at the same slice-context and hence the same roofline cost. Within a cost-homogeneous bucket avoidable_exec = total_exec * (builds - necessary)/builds is exact; the buckets of one DAG node aggregate back to a per-label AvoidableNode (what the visualizer joins on by hash->sig). Verified: buckets are homogeneous (count*last == total*frac per bucket), all arms bounded <= 100% (C60 external-occ 390% -> 0.005%), and the giant term reads 77%, matching an independent dependent-mode analysis (78%). NodeCost now accumulates total_exec/total_flops; min/max/last are kept only for the SEQUANT_AVOIDABLE_DEBUG homogeneity dump.
Redefine the per-value avoidable-recompute metric: it is now measured in FLOPs
against the batching-free (unlimited-memory) ideal -- the arithmetic the batched
replay repeats beyond building each value once at full extent -- rather than in
roofline exec against a within-scheme "necessary" reference.
Two problems with the exec-weighted metric drove this:
- roofline exec is nonlinear in slice size, so one value's differently-sized
builds spanned ~257x; pricing avoidable as count x a single per-build exec
exceeded the whole replay's exec (avoidable_time > 100%). The prior
cost-homogeneous slice-context bucketing removed the >100% but stayed
exec-weighted;
- referencing "necessary = distinct slices the scheme produces" is circular --
it scores an un-hoisted value's per-block rebuilds as necessary, so it cannot
see the recompute hoisting exists to avoid.
FLOPs is linear in extents, hence additive across slices: disjoint per-block
slices that tile a value sum to exactly full_flops (0 avoidable), while a value
rebuilt full per block sums to N*full ((N-1)*full avoidable). So avoidable =
max(0, total_flops - full_flops) per value, bounded in [0, dryrun_flops] by
construction, needs no slice-context bucketing, and answers "what does batching
cost vs. infinite memory". NodeCost drops to {builds, total_flops, full_flops};
CostProfile.avoidable_exec -> avoidable_flops, avoidable_time() = avoidable_flops
/ dryrun_flops.
Witnesses re-baselined (nterms=55, FLOPs): occ-veto 1.8/6.5/15.4%; the extmode
witness shows external-occ (~1.95%) and contracted-occ (~1.97%) essentially
equal -- external-mode batching is NOT a recompute fix on the C60 forest (its
original conclusion, now with honest magnitudes). The [cost_profile] giant reads
77.8%, matching an independent dependent-mode analysis (78%).
The batch-variant caching veto in cache_manager had two disjuncts: (a) a node whose own batched_here() carries a Contracted, batchable mode FREE in its own result, and (b) a non-empty cross-occurrence lifetime mask. Disjunct (a) is structurally dead: a Contracted mode is summed AT the node, so it can never be free in that node's result, and post-role-split a free index is stamped External, never Contracted -- so the condition never holds (the occ-veto test's [veto-reach] probe read 0, structurally, not by forest accident). It only ever guarded a malformed emission. Remove disjunct (a) and the `is_batchable_contracted_index` parameter from the cache_manager factory (the only functional caller, build_dryrun_cache, drops the arg; no production caller passed it), the `CacheConfig::is_batchable_index` field and the cost_profile() overwrite that fed it, and the now-obsolete [veto-reach]/[veto-hazard] probes plus the disjunct-(a) sub-test. Disjunct (b) (the cross-occurrence lifetime-mask veto -- the load-bearing F1 correctness guard) is unchanged. Behavior-preserving: [cache_manager] (200), [lifetime_mask] (76), [dryrun], and [eval] (TA production path, 457) all green. Does NOT touch BatchPolicy::is_batchable_contracted_index (the batching decision) or BatchPolicy::is_batchable_index() (the eval accept union) -- both stay.
Design note reframing batched-eval cache placement as register allocation. Three identities -- value (hash), instance (use-site), cell (a materialized copy serving a subset of instances). A value's instances partition into cells; perfect CSE = one cell/value, no CSE = one cell/instance, and the peak budget chooses the granularity in between (a partial un-CSE / materialization DAG). Cell identity = (value, home-scope, split-index): home-scope is the loop level (the axis batching adds), split-index names a same-scope peak split (the RA live-range-split rename). Placement is register allocation + loop-invariant code motion + rematerialization: hoisting a shared value lengthens its live range (peak) to save recompute; slicing adds partial-hoist granularity. Objective: minimize recompute (the rational, batching-aware reuse count W-1 times build cost) subject to the whole-forest peak profile <= peak_threshold. Peak is a placement (post-CSE, whole-forest) constraint, not a factorizer one; cost_profile()'s replay peak is the detection safety net, and a peak that survives full splitting is factorization-inherent. Includes a prior-art section (rematerialization/checkpointing -- Checkmate; electronic-structure space-time tradeoff -- Cociorva/Sadayappan PLDI 2002; register allocation; pebble games), four worked cases, and open items (group-scoped cache keying, the greedy split move, per-placement footprint, W's fixed point).
O1 (cell keying) resolved as a router + dumb stores, not a wider cache key. Add
§7a "Runtime realization": one value-keyed store per (home-scope, split-index)
-- the cache stays TreeNode-keyed unchanged -- plus an explicit router
{value, use-site} -> (home-scope, split-index) that is the placement pass's
output and replaces the implicit parent_ fall-through search. Reads route via
the map then reuse the EXISTING Enter-stage slicer, (use-scope - home-scope)
INTERSECT carried(N), fed the home scope directly instead of via hops; default
{value} -> (home, 0) is byte-identical. Standardize terminology on "home scope"
(= the code's "lifetime scope" = store scope; consumer's is "use scope"). Update
§4, §9, and O1 accordingly; residual O1 sub-items are the use-site/occurrence id,
the parent_/hops audit, and the naming standardization.
Add §7b: the placement pass as a register-allocation spill loop. Seed = perfect CSE (recompute-minimal, peak-maximal); walk up the recompute axis to walk down peak until peak <= threshold. Objective and constraint are exactly cost_profile()'s avoidable_flops and peak_bytes -- no new measurement. Moves: SHRINK (slice a carried mode a cell holds full -- the existing external-slice / node_level_placement, now driven off the true whole-forest peak) and EVICT (delay/un-hoist an invariant cell held idle, or split a long-lived cell's instances into short-lived groups -- the new CSE-aware move the per-term DP cannot see). Greedy: candidates = cells alive at the binding peak point, prefer free shrinks then max ΔPeak/ΔRecompute (the spill metric), apply, incrementally re-cost, repeat; terminate on fit or on a factorization-inherent peak. Residual sub-items O2a (incremental profile update), O2b (per-move estimator/lookahead), O2c (subsume vs run-after the DP external-slice pass).
Clarify §7b: O2 runs after the per-term min-time factorizer and takes the factorization AND batch-loop assignments (batched_here) as FIXED, deciding only the whole-forest eval/placement strategy (home-scope + router); it never adds, removes, or re-assigns a batch loop. Reframe "shrink" from "slice a carried mode" to "re-home a cell into an EXISTING carried loop" -- a placement choice on the fixed nest, not a batching change; deciding to batch an un-batched mode (adding a loop) is the factorizer's lever. Split the termination boundary into two non-O2 failure modes: factorization-inherent (a single intermediate > budget) vs. re-batch-needed (fixed batching left placement too little room, e.g. a shared cell needing slicing on a mode no single term batched) -- both detected via peak_bytes and fed back, giving the structure factorize+batch -> O2 place -> if infeasible re-batch.
Add §7c. Cell footprint is home-relative: a carried mode is sliced (block extent) iff its fixed batch loop encloses the cell's home, else held full -- the existing moment-aware memsize with home-relative extent overrides, so O2's shrink ΔPeak is just the footprint delta. The peak profile is max weighted-interval overlap: each cell is a [first-use, last-use] interval (from the router's use-sites + the static schedule order) weighted by footprint; peak = max over static points of the sum of live cells' footprints (a sweep line), and the argmax is O2's binding peak point. Because it SUMS co-resident live cells it corrects today's peak_bytes = max(scratch, cache) under-count (a lower bound per §1); the replay stays the oracle (must sum, not max, co-residency). The weighted-interval form updates incrementally under an O2 move (feeds O2a). Residual O3a-c: the sweep structure, the summed- co-residency replay oracle, composite/proto sizing.
Add §7d. Define home_scope(value) = deepest scope enclosing the loops of (sliced_modes ∪ demoted_external_modes). sliced_modes is the cross-occurrence meet (max-reuse upper bound); the demotion fold adds the External batched_here stamps the meet demoted (has_demoted_external) -- occurrences bind them to incompatible blocks, so the value can't be a single full value above those loops and its home must be inside them. The fold is exactly what unifies the current cache-veto-vs-has_demoted_external disagreement into one authority both the cache and the runtime read. Per-block is temporal (one external-loop-homed cell re-instantiated per iteration), so no split-index -- that stays reserved for O2's peak-driven same-scope splits. Structural and computed from the meet before O2, which only lowers homes further for peak; consistent with W (the demoted mode is free tiling). Residual O5a-b: confirm the exact signal / edge cases and the seed router construction. Also tie O6 to §7b's two failure modes.
O4 (W's computation order) is not a fixed point: W is a function of the current placement, well-defined at the home_scope seed and re-costed incrementally per O2 move -- seed-then-refine, subsumed by §7b/§7d. O6 (feedback) scoped to a minimal detect-and-report step (surface the binding cell + failure mode so a schedule fails loudly, not silent OOM), with the re-batch/re-factorize hint as a follow-on that the detect step precedes. All major open items (O1-O6) now designed or resolved; the spec is design-complete.
Phased plan for the placement-as-register-allocation design. Phase 1 (detailed, bite-sized TDD) corrects cost_profile()'s peak_bytes from max(scratch, cache) hwmarks to the instant-resolved co-resident SUM across the scope chain (spec 7c/O3b) -- adds CacheManager current_residency()/chain_residency(), threads the chain sum into note_working_set, simplifies the fold, and re-baselines the documented-RED peak figures from measurement. Phases 2-5 (router+home_scope seed, static peak sweep, the O2 greedy, feedback) are a roadmap, each a future plan. Global constraints: no en-dashes, clang-format, byte-identical perfect-CSE default, replay stays the peak oracle.
…x (Phase 3b review fix)
…ity classify; rename the pass O2 -> remat (Phase 4a T2)
…n-regression (Phase 4a T3)
…ing); novelty is the CSE-forest scope + meet; note forest-level Pareto-DP borrow
…cted_modes (Phase 4b-1 T1) Drop the External-only filter in stamp_lifetime_masks so sliced_modes is the cross-occurrence meet of ALL batched modes on a node's result slots; delete the per-occurrence contracted_modes bolt-on end-to-end (EvalExpr field/accessors, NodeBatchAnnotation, cost-model emission, place_at_this_level in_union/ residency_all_outer, schedule_dump). The unified meet subsumes contracted_modes and corrects its latent per-occurrence aux under-meet. The has_demoted_external veto stays (Phase 4b-3). seed_residency stays (retired in Phase 4b-1 T2). Correctness-preserving: full unit suite green (313474 assertions); the C60 witnesses are byte-identical (occ-veto 38897.4/6047.4/525.9 GB, External-occ=244; extmode 6026.0/5999.7 GB) with the 2 pre-existing documented-red targets unchanged. MPQC CCk validation (cross-repo) is the final energy gate before merge.
…des (Phase 4b-1 T2) After T1 unified sliced_modes to the all-batched-modes meet, the separate seed_residency field (Phase 3a) is identical to sliced_modes. Delete stamp_seed_residency + EvalExpr::seed_residency_ (+ accessors); home_scope now returns sliced_modes(); peak_profile's linearize_rich calls stamp_lifetime_masks. Byte-identical for the peak profile / remat (anchor 481600, end-to-end 16128->160 unchanged); the [seed] tests converted to drive stamp_lifetime_masks/sliced_modes, and the 3 tests that only validated the two-field separation (now vacuous) removed with their dead helpers.
…_boulevard The eval forest is flattened to a linear static-point timeline; the thin schedule is the 'path', the rich per-value schedule (more fields per cell) the 'boulevard'. Pure rename; behavior unchanged (303 assertions green).
For each cell whose remat home_modes differ from its seed home, emit one
PlacementRouter override per distinct occurrence key of that value -> the shared
HomeTarget{home_modes,0}. Value hash links a RematResult cell to its forest
nodes; occurrence keys (bliss) are computed lazily here, moved cells only.
Additive: zero runtime wiring; empty router when nothing moved.
…nce routing - rename remat_to_router's 'remat_cells' param to 'final_cells' (it shadowed the sibling remat_cells() free function in the same header). - extend the giant test to G's SECOND occurrence: on this fixture the two occurrence keys are IDENTICAL (G's batched o_1/i_1 are its result slots, absent from its subtree leaves A/B, so the batched-slot coloring cannot distinguish them), and both resolve to the shared home. The genuine two-distinct-keys partial-overlap case needs a fixture whose value subtree carries the batched modes -- documented as deferred.
…me the synthetic-fixture note
The earlier giant fixture's two occurrence keys coincided only because it is
SYNTHETIC (G{o_1;i_1}'s declared slots are inconsistent with its A/B subtree, so
occurrence_key -- built from the subtree leaves -- cannot see o_1/i_1). In a real
eval tree an intermediate's batched result modes ARE genuine subtree indices. New
test uses a CONSISTENT fixture (V{i_1;i_2}=A{i_1;x_1}*B{x_1;i_2}, sliced {i_1,i_2}
under P1 vs {i_1} under P2): the two occurrences then have GENUINELY DISTINCT keys,
and remat_to_router emits both -> the SAME HomeTarget (the partial-overlap /
meet-home case, design 10.1). Reframed the giant test's comment accordingly.
…ite-default policy (section 5)
… T2 MPQC pre-pass, T3 CCk validation)
…ed (Phase 4b-3 T1) Placement in place_at_this_level is now purely router-override-or-seed: an order-aware, residency-all-outer node is hoisted to its seed home unless the remat router (populated by the MPQC pre-pass, a later task) overrides it. A value cached at its seed home is the same value the descended path produced (the batched slice-on-use slices it per block), so this changes PLACEMENT only, never the numerical result. Adds the router-vs-store-seam consistency invariant test (the pre-pass keys and home resolution == the runtime store seam), re-baselines the batched- external placement witness (the demoted carrier now hoists once, result still exact), and re-baselines the two dry-run witnesses' NOT-TRUSTED diagnostic figures (demoted external giants now hoist: modelled avoidable recompute falls, modelled peak rises).
Add an OWNING backing for the placement router: a shared_ptr member (owned_router_) plus adopt_placement_router(), which stores the router and points the existing non-owning placement_router_ at it. A router built by a pre-pass (the remat placement pass) is a build-site local; a CacheManager returned by value from such a builder must carry the router alive with it. shared_ptr keeps CacheManager copyable and type-erases the forward-declared PlacementRouter's deleter, so the incomplete type in the member declaration is fine. This is the seam the MPQC remat pre-pass wires into next: it builds the router on the real EvalExprTA nodes and hands it to the returned cache via adopt_placement_router.
Add eval::ScheduleSink {ostream* os; bool fired;} + CacheManager::
set_schedule_sink / schedule_sink() (parent fall-through, like placement_router).
The two cache-accessible SCHEDULE_RUN_EVENT emit sites in evaluate() (the
finish_phase_b Build/Fetch choke point and the cache() Store/Access/Release
hook) now write to the wired sink instead of a process-global getenv->cerr, so a
caller can capture ONE evaluation's batched schedule to its own destination and
fire once (the `fired` latch). No sink (the default) => no dump, byte-identical
(eval/eval_ta/cache_manager: 687 assertions unchanged). The SCHEDULE_RUN_GROUP
marker inside make_batched_custom_evaluator (no cache in scope) stays on the env
var for now; it is threaded to the sink in Phase C when the DAG clustering
consumes it. End-to-end capture is validated by the MPQC w8 dry/wet dump (Phase B).
…sink Add an optional ScheduleSink* param (default null, byte-identical) set on the internal replay cache alongside the router. Lets MPQC capture the DRY replay's batched schedule to a file and diff it against the WET run's schedule (same sink type) for structural equivalence.
…term reset) The replay's per-term cache.reset() dropped non-persistent scratch between summands, so a value shared across summands was REBUILT in each -- over-counting recompute (and mis-estimating peak) vs any real solve, whose cache spans the whole iteration and reuses cross-summand values, evicting only by the lifetime mask stamped over the whole forest. Remove the reset (and the short-lived reset_between_terms flag): the replay now models the real shared-cache scope, so the dry-run schedule matches the wet run's node-for-node on build counts and loop scope (the dry-vs-wet equivalence path, doc 2026-08-05). Re-baseline the two [.] witnesses this shifts: - dryrun-occ-veto: aux+occ documented-RED peak 1198.5 -> 1431 GB (no per-term reset holds more cross-term scratch co-resident; the intentional <100 GB research-target RED is unchanged). - dryrun-extmode-avoidable: the contracted arm's ~1.97% avoidable was a PER-TERM -RESET ARTIFACT (cross-summand rebuilds a real solve never does); with the faithful cache both arms are near-zero (~0.005% / ~0.025%), so the earlier "external below contracted on recompute" finding was reset-induced -- the real distinction is peak (external ~12481 vs contracted ~6133 GB). Narrative + checks updated to "both near-zero / match". Other slow witnesses (extmode-sweep, occ-sizing, water20-overcompute, perfcost) pass unchanged; occ-recognize's failure is pre-existing (batchable_mode_list recognition gate, untouched by this change).
The DryRun mode_batches made UNIFORM target_batch_size blocks over the dense extent, but a real (tiled) backend partitions a batch axis into WHOLE-TILE batches -- so when a tile is coarser than target_batch_size the dry-run's batch COUNT (hence its recompute) diverges from the wet run (w8-f12: aux Κ is 4 tiles of ~168 but aux_target_size=64 -> wet 4 batches, dry 11). Add SizeRegime::space_slice_extents: an OPTIONAL per-space batch partition (the element extent of each realized slice) the CALLER supplies. mode_batches reads it directly (accumulated to [lo,hi)); empty => the uniform fallback (byte-identical; [dryrun]/[cost_profile] 122 assertions unchanged). The dry-run backend does NOT know how the partition was derived, so any backend model plugs in without touching eval internals. batch_slice_extents_from_tiles is a ready-made converter for a TILE-based caller (replicates the wet mode_batches_of_trange1 grouping).
mode_batches applied a caller-supplied batch partition only when the mode was at full extent (slice_sum == extent). A NESTED batch call -- the axis already narrowed by an outer batch (ExtentOverrides) to one of its own slices -- failed that strict guard and fell through to uniform target_batch_size blocks, re-slicing an already-atomic sub-range (e.g. a 168-element aux tile into ceil(168/64)=3 blocks, which then cascade). The wet backend never does this: mode_batches_of_trange1 reads the sliced operand's real TiledRange1, so a single-tile sub-range is one atomic batch. Emit instead the PREFIX of partition slices that sums to the requested extent: the outer call (extent == full axis) yields the whole partition, a nested call yields the tiles spanning that sub-range, so nested batch counts match the wet backend's per-tile atomicity. If the extent does not land on a partition boundary, fall back to uniform blocks as before. Makes the CCk dry-run realize the byte-for-byte batched schedule the wet run does: w8-f12 aux-batched PNO-CCSD is now structurally identical (build-count + loop-scope) across all 320 shared nodes, vs 16 divergent aux-loop nodes before. Behavior-neutral when no partition is supplied (slices empty -> uniform path unchanged); [dryrun]/[cost_profile]/[eval] unit witnesses unchanged (668 assertions).
…gine The batched evaluator re-enters the recursive evaluator for every batch block (the external-mode scatter and the contracted-mode contraction each re-evaluate a node's subtree per block) and for every hoisted loop invariant. Those re-entries called evaluate() itself, so evaluate() was not the single outermost call -- it was re-invoked from inside the batched machinery, which makes the control flow hard to reason about. Rename the recursive engine to evaluate_impl and make evaluate a thin redirect to it. Point the internal re-entries (the layout wrapper's engine call, the scatter and contraction per-block re-evaluations, and the hoisted-invariant build) at evaluate_impl. Add an empty-cache evaluate_impl overload for the hoist build (which runs on a fresh cache). Public callers and the range/layout/empty-cache/symmetrize wrappers are unchanged, so this is behavior-neutral: `evaluate` now denotes the top-level call and the whole batched recursion is contained in `evaluate_impl`. No functional change; SeQuant [eval]/[dryrun]/[cost_profile] unit tests pass and the CCk dry-vs-wet batched schedule is unchanged (structurally identical, w8-f12 aux+occ).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multimode batched evaluation of factorized coupled-cluster equations
Adds cost-model-driven multimode batching to the SeQuant evaluator so
large-system CSV/PNO-CC residuals can be evaluated without forming their
largest transients whole. Six squashed commits (dry-run backend, optimizer,
evaluator, supporting core, tests, docs).
What it does
-> scattered into disjoint slices) and contracted (DF aux, summed ->
accumulated). Loops nest external-outside-contracted.
DenseTimeSpace): minimizes flops withpeak_thresholdas a ceiling; role-split (contracted/external) batchability;order-aware placement over the combined nest.
(a cached intermediate fetched from an outer scope is sliced to the current
block) decouples correctness from placement; per-level placement driven by
a per-canonical lifetime mask (cross-occurrence proto-aware meet) unioned
with contracted residency; iterative (stack-safe) tree traversal.
C60 PNO-CCSD dry run (55-term residual, aux K@256, occ@8, 100 GB budget)
The DP selects the same factorization regardless of what is batchable
(flops are unchanged); batching only slices modes to lower the peak. The
roofline-time column moves because a giant intermediate executed whole is
memory-bound (
machine_balance x traffic) but compute-bound when sliced -- thecache-blocking win of the same schedule, not a cheaper one. Recompute overhead
(
avoidable_time) is 1.8% -> 6.5% -> 39.8% as slicing gets more aggressive.Validation
[eval]449,[lifetime_mask]76,[optimize]628 assertions green;OFF (order-blind) path byte-identical.
events) matches unbatched to < 1e-9, within the 1e-7 precision, no aborts.
Follow-ups (non-blocking, from the final review): dedup the proto-expansion
helper; add a real-forest hidden-tag hash-regression test; revisit the
stamp_lifetime_masksconst_cast.