Cost and Factorization Analysis Utility - #577
Open
ajay-mk wants to merge 20 commits into
Open
Conversation
A standalone utility that estimates the cost of evaluating a serialized tensor equation. Reads a JSON driver (space sizes + equation files), strips the (anti)symmetrizer, flattens, then per summand optimizes and binarizes once, reusing that tree for the intermediate catalog, cache simulation, and --dump_tree, mirroring MPQC's SeQuantEngine. Emits a Markdown report on largest intermediates, most expensive contractions, shape census, peak storage, FLOPs, and a gated cache simulation. Includes a README and CCSD/DF example drivers.
Diff generated reports against frozen *.md.expected via run_tests.cmake (volatile lines stripped): a spin-orbital CCSD R2 (a Sum of terms, with a --dump_tree check) and a single density-fitted product (the non-Sum path).
Krzmbrzl
reviewed
Jul 9, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new standalone cost_analysis utility under utilities/ that reads a JSON driver, optimizes/binarizes SeQuant equations, and emits a Markdown report (plus optional tree dumps) along with reference-style CTest coverage.
Changes:
- Added the
cost_analysisexecutable with JSON-driven configuration, equation parsing, optimization/binarization, cataloging, and optional cache simulation. - Added Markdown report generation and accompanying documentation/examples with frozen
*.md.expectedfixtures. - Integrated the tool into the build and added CTest-based golden-file regression tests.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| utilities/cost_analysis/run_tests.cmake | CMake script to run the utility and compare generated reports to expected fixtures (with volatile lines stripped). |
| utilities/cost_analysis/report.hpp | Declares report-writing and tree-string serialization helpers. |
| utilities/cost_analysis/report.cpp | Implements Markdown report formatting, table generation, and tree stringification. |
| utilities/cost_analysis/README.md | Documents the driver schema, workflow, report contents, and tests. |
| utilities/cost_analysis/examples/df_r1.md.expected | Golden expected report output for DF R1 example. |
| utilities/cost_analysis/examples/df_r1.json | JSON driver for DF R1 example. |
| utilities/cost_analysis/examples/df_r1.inp | Serialized equation input for DF R1 example. |
| utilities/cost_analysis/examples/ccsd_r2.md.expected | Golden expected report output for CCSD R2 example. |
| utilities/cost_analysis/examples/ccsd_r2.json | JSON driver for CCSD R2 example (includes dump_tree). |
| utilities/cost_analysis/examples/ccsd_r2.inp | Serialized equation input for CCSD R2 example. |
| utilities/cost_analysis/examples/.gitignore | Ignores generated reports, filtered files, and dump-tree outputs. |
| utilities/cost_analysis/cost_analysis.hpp | Defines shared config/result data structures between analysis and reporting. |
| utilities/cost_analysis/cost_analysis.cpp | Implements the CLI entrypoint, JSON parsing, context setup, analysis pipeline, cache simulation, and report emission. |
| utilities/cost_analysis/CMakeLists.txt | Adds build target and CTest registrations for the new utility. |
| utilities/CMakeLists.txt | Wires utilities/cost_analysis into the utilities build. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The generated timestamp changed every run; only the volatile-line stripping in run_tests.cmake kept the reference tests deterministic. Remove it (and the now-unused <chrono> include). Keep the SeQuant version/hash line, still stripped before diffing. Reference fixtures now carry a <version> placeholder instead of a frozen timestamp+hash.
ajay-mk
marked this pull request as ready for review
July 10, 2026 21:10
The input list is the equations to analyze; 'results' collided with the tool's computed output. Rename the JSON key, EquationSpec, and Config field so input equations read distinctly from output results.
Mirrors the existing min_repeats guard: a negative top_n wraps to a huge size_t, and a negative max_footprint silently disables the cache budget.
The driver chdir's into its own parent, so a relative output.path resolves there rather than the invocation directory; print the absolute path in the confirmation line and document the resolution rule. Also document the registry-default fallback for unlisted sizes, correct the report layout description (Cache is one pooled section, not per-result), and trim the CSV/PNO note.
Krzmbrzl
approved these changes
Jul 15, 2026
Krzmbrzl
left a comment
Krzmbrzl
left a comment
Collaborator
There was a problem hiding this comment.
I'm wondering whether there is potential for merging with the external interface. I am currently in the process of rewriting its core design to be processing-step oriented and in this design creating this cost analysis could be a useful thing to be capable of.
Plus, this would have the advantage of a unified JSON driver input syntax.
That's something for the future though and shouldn't hold up this PR.
Member
Author
I think that is a good idea. I initially considered doing that, but wasn't sure if we would want to merge this utility into SeQuant. I wrote it mainly to see how factorization changes when we apply DF/THC to the equations, but turned out to be quite useful. |
Collaborator
|
Yeah, I think it wouldn't really fit with the current structure of the external interface but with the refactor it should be relatively easy to pull something like this off 👍 |
Replace the run_tests.cmake -P wrapper with a generate, sed (strip volatile line), diff multistep process.
Krzmbrzl
reviewed
Jul 15, 2026
Add --omit-revision to drop the volatile git-revision line, and end the report with a single trailing newline (leading section separators instead of trailing ones) so it is byte-stable. The tests then use plain cmake -E compare_files instead of sed/diff, making them portable. The tree dump is diffed against a new R2.tree.txt.expected fixture.
Member
Author
|
Another round of Claude review found a few issues, |
spbasis was a bare string comparison, so any typo silently selected spinfree with no diagnostic; validate it like the sibling field option. volatile_weight had no validation at all -- a non-positive value makes the DP tie-break's cost negative for volatile-containing subsets, which the optimizer would then treat as cheapest, silently choosing the wrong schedule. Reject it, matching MPQC's own SeQuantEngine validation of the same option.
res.largest_mem and res.peak_storage compared via AsyCost::operator<, which orders by symbolic polynomial degree, not actual size at the driver's configured sizes. A lower-degree intermediate can be numerically larger than a higher-degree one at those sizes, so the reported largest/peak could silently be the wrong term -- while the tables printed right below them already sort by actual MB. Compare by .ops() instead. The library's own min_storage() has the same symbolic-max issue internally (it's used for peak_storage), so add a local numeric_peak_storage that reimplements its traversal with a numeric max; fixing min_storage() itself is out of scope here since it's shared library code. Updates the df_r1 fixture: its peak storage was under-reported at 6.8420 MB by the old comparison; the correct value is 7.0801 MB.
Only the ofstream open was checked; std::ofstream::operator<< doesn't throw on write failure by default, so a disk-full or I/O error mid-write would silently produce a truncated report with exit code 0. Close the stream and check it before declaring success.
- setup_context's summary comment said field is applied per-space; the code six lines below applies it to every registered space. - space_signature's 'X' bucket comment said size-grouping in the shape census only happens with multiple aux spaces registered; X also catches any other non-hole/non-particle space (e.g. frozen-core), so grouping can happen with one aux space or none. - report_expensive's sort comment had an unfinished trailing fragment. - README didn't document that machine_balance/fast_mem_elems are only forwarded when volatile_leaf is set and cache.enabled is true, or that toggling cache.enabled can change the chosen contraction order. - README described Cached/Persistent as separate counts; Persistent is a subset of Cached, not disjoint from it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cost and Factorization Analysis Utility
Adds
utilities/cost_analysis, a standalone tool that estimates the cost of evaluating a tensor equation without running it. It reads a serialized equation, optimizes and binarizes it, then writes a Markdown report on the largest intermediates, the most expensive contractions, peak storage, operation counts, and cache reuse.Inspiration is from MPQC's
SeQuantEngineclass and the several knobs it exposes.What it does
<head> = <rhs>in SeQuant serialization V1), plus context (spin basis, field, index-space sizes, factorization spaces).deserialize→ strip leading (anti)symmetrizer →flatten→optimize+binarize→ catalog the tree's internal nodes → cache simulation across all results.contractions, a Shape census by O/V/X signature, and a Cache table.
Usage
All driver knobs and their defaults are documented in the utility's
README.md.Tests
Two reference tests (
ctest -R cost_analysis_) run the tool onexamples/and diff against frozen*.md.expected.Not supported right now
AsyCostis not proto-index aware; this utility should work onceAsyCostis extended.Example report:
EOM-CCSD(2h2p) R2 (53 terms; i=10, a=38)
Terms: 53; distinct intermediates: 94; reused: 9; largest: 8.3728 MB; peak storage: 40.1953 MB.
Total operations (symbolic): 4i^2a^4 + 18i^3a^3 + 26i^4a^2 + 2ia^4 + 30i^2a^3 + 104i^3a^2 + 26i^4a + 4ia^3 + 55i^2a^2 + 4i^3a + 10*i^2a
Largest intermediates
Most expensive contractions
Shape census
Cache (gated, volatile leaf = "R", min_repeats = 2)