Repository navigation
FIX: stop flaky wall-clock margin in hierarchy budget-exhaustion test - #3680
Open
Pushpak Siva Sai (Pushpak731) wants to merge 1 commit into
Open
Pushpak Siva Sai (Pushpak731) wants to merge 1 commit into
Pushpak Siva Sai (Pushpak731) wants to merge 1 commit into
Conversation
The 0.05s assertion had no tolerance for runner contention and flapped on main (1.73s on a contended ubuntu runner in run 41014283717, an 8% overshoot on macOS in run 40810357178). The halt itself is already proven structurally — exact node count of 1, the truncated flag, and the exact aggregate — and wall clock cannot guard the timing part at this scale: the bounded traversal takes ~6ms and a full build of all 1,000 nodes only ~20ms, so contention noise (~1.7s observed) is roughly 85x the full-work cost and no tight bound separates 'immediate' from 'quadratic'. Replace it with a generous 5s ceiling that still fails fast on a hung traversal instead of waiting for the job to time out. Closes microsoft#3671.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
test_hierarchy_traversal_halts_immediately_on_budget_exhaustionasserted the bounded traversal completes in under 0.05s of wall clock. That margin has no tolerance for runner contention and has flapped on main repeatedly (#3671):Expected <0.05s, got 1.7288sWhy the bound can't be saved by choosing a better number
I measured the two regimes the test tries to separate, on the same 1,000-root fixture the test builds:
max_nodes=1, current behavior)max_nodes=1000)Even building every node costs ~20 ms, while contended runners have measured 1.7 s for the same call — the noise is ~85x the full-work cost. At this scale no wall-clock bound can both tolerate CI scheduling noise and catch a quadratic regression, so the timing assertion cannot be the regression guard.
The fix
node_count == 1), is flaggedtruncated, and keeps the exact aggregate (overall_counts.completed == 1000). A traversal that ignored the budget cannot produce that projection.Validation
tests/unit/backend/test_scenario_hierarchy_parity.py— 84 passed.ruff check/ruff format --checkclean.Closes #3671