Skip to content

BUG test_hierarchy_traversal_halts_immediately_on_budget_exhaustion fails intermittently on CI (wall-clock bound) #3608

Open

Description

Description

tests/unit/backend/test_scenario_hierarchy_parity.py::test_hierarchy_traversal_halts_immediately_on_budget_exhaustion (added in #3167) times ScenarioProgressReadModel.build_scenario_hierarchy with 1,000 roots and max_nodes=1, then asserts elapsed < 0.05. Since it landed, build_and_test fails intermittently on unrelated PRs, always on this assertion:

FAILED tests/unit/backend/test_scenario_hierarchy_parity.py::test_hierarchy_traversal_halts_immediately_on_budget_exhaustion - AssertionError: Expected <0.05s, got 1.5995s

Runs on 2026-10-28 that failed only on this test, with measured times between 1.60s and 2.25s:

Branch Job
romanlutz-gcg-b4-dataset-adapter main-job
maint/3166-p311-03-codeattack (juliosuas) main-job
fix-braille-letter-sign (barkha1research-ai) main-job
fix/fairness-bias-response-message (#3556) ubuntu-latest, 3.14, dev
fix/dataset-max-examples-at-least-one (#3592) windows-latest, 3.12, dev
fix/reject-nan-temperature-top-p (#3606) windows-latest, 3.14, dev

The push run on main (1e6f925) passed the same jobs.

Why the call isn't slow

On a local machine the call takes about 8 ms on main and scales linearly (4,000 roots in about 34 ms). The test also passes when the whole file runs in one process (84 passed). A 3M-object live heap, with GC on or off, didn't change the timing.

The 1.6s stalls therefore come from the CI worker, not from the traversal. make unit-test-junit runs pytest -n 4 --dist=loadfile, so this test shares a worker with whatever ran before it on a loaded runner. A 50 ms wall-clock bound can't tell that apart from a regression.

Proposed fix

Keep the intent (the traversal stops at the node budget instead of walking all 1,000 roots) and assert the work done rather than the time taken. Spy on ScenarioProgressReadModel._map_hierarchy_node and assert it's called exactly once. With both budget checks in build_scenario_hierarchy removed, this assertion fails with 1000 == 1, so it still catches the regression the timing was meant to catch. I'll open a PR with this change.

Current coverage and overlap, October 31, 2026

Assigned to ShivamSharma43 and already covered by open #3609. #3671 reports the same assertion, with competing #3680 and #3688 also editing this test. Coordinate one fix across the existing issues/PRs; these are not separate hierarchy defects or new GUI milestones.

Keep the output/truncation/exact-total checks, but preserve a deterministic work-bound regression guard as well. A clipped one-node output alone does not prove that extra roots were not visited or materialized before clipping. #3609 proposes a mapping spy; #3680 widens the timing threshold, and #3688 removes it without adding a work-count check. This audit inspected those proposals but did not run their tests or choose/merge a fix. Original run measurements above remain the reporter's evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions