fix(tests): WAL-pinning for all inline NullRunRuntime creations#63
Merged
Conversation
The test_runtime fixture in test_protect_branches.py built a real NullRunRuntime(api_key, _test_mode=True) without going through the mock_api conftest. The runtime's Transport.start() calls _replay_from_wal() which reads /tmp/nullrun.wal (the default NULLRUN_WAL_PATH fallback). If a previous test run in a different Python version (3.10 or 3.12) had persisted a non-empty WAL, the 3.11 worker would replay those events to a real HTTP endpoint, get HTTP 401, and the fixture would fail at setup with NullRunAuthError: nullrun.breaker.exceptions.NullRunAuthError: Invalid API key This bit CI on 2026-07-11 (run 29156199607): tests 3.10 + 3.12 passed, test 3.11 failed with that error in the fixture setup of test_enforce_sensitive_tool_dict_with_fallback_fail_open. The 3.10/3.12 runs cleared the global /tmp/nullrun.wal by reading it first, so 3.11 picked up the next writer. Order- dependent; flaky on the matrix. Pin NULLRUN_WAL_PATH to a tmp_path-scoped file so each test session reads its own fresh empty WAL. Resolves the flake without touching SDK source (no production code change). Verified locally: - pytest tests/test_protect_branches.py -> 43/43 pass - with pre-seeded stale /tmp/nullrun.wal, the previously failing test now passes. No public API change. No SDK_MIN_VERSION bump. Backends on 1.0.0 keep working unchanged. Recommended: 0.13.6 (no version bump needed for a test-only fix).
Follows up on commit 41a16f7 which pinned NULLRUN_WAL_PATH for the test_runtime fixture only. Other tests in test_protect_branches.py / test_runtime_branches.py / test_toolbox_langgraph.py build NullRunRuntime inline (no fixture) and were still picking up a stale WAL from a previous test run, causing HTTP 401 `NullRunAuthError` in 3.12 (CI run 29158094827, job `test (3.12)`). This commit: 1. Adds a shared `make_test_runtime` factory fixture to conftest.py that pins NULLRUN_WAL_PATH to tmp_path, stubs _do_flush / _do_flush_locked / _client, and resets the singleton around the factory. 2. Replaces 4 inline `NullRunRuntime(api_key=..., _test_mode=True)` calls in test_protect_branches.py with `make_test_runtime()`, including: - test_protect_async_kill_re_raises_WorkflowKilledInterrupt - test_get_protected_runtime_falls_back_to_get_runtime 3. Patches the local _make_test_runtime / _make_runtime_with_mocked_auth helpers in test_runtime_branches.py to set NULLRUN_WAL_PATH per-call (via tempfile.mkdtemp) before constructing the runtime. 4. Extends the autouse _test_runtime fixture in test_toolbox_langgraph.py to take tmp_path and pin NULLRUN_WAL_PATH, matching conftest::make_test_runtime. Verified locally on 3.11: - pytest tests/ -n auto: 1219 passed, 1 failed, 7 skipped (1 failure: test_actions.py::TestPauseAction ::test_is_paused_respects_cooldown — pre-existing flake on master, NOT introduced by this commit; verified by git stash + repro on bare master) - ruff check src/: all checks passed - mypy src/: success, no issues in 34 source files Public API unchanged. No SDK_MIN_VERSION bump. Backends on 1.0.0 keep working unchanged. Recommended: 0.13.6 (no version bump needed).
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follows up on #62 (commit 41a16f7) which pinned
NULLRUN_WAL_PATHfor thetest_runtimefixture only. Other tests intest_protect_branches.py/test_runtime_branches.py/test_toolbox_langgraph.pybuildNullRunRuntimeinline (no fixture) and were still picking up a stale WAL from a previous test run, causingNullRunAuthError: Invalid API keyintest (3.12).CI run 29158094827 —
test (3.12)failed ontest_protect_async_kill_re_raises_WorkflowKilledInterrupt. Thefail-fast: truematrix strategy stopped 3.11 mid-run, thencoveragealso failed because it depends on the test job.Root cause
test_protect_async_kill_re_raises_WorkflowKilledInterruptbuildsNullRunRuntime(api_key="test-key-12345678", _test_mode=True)inline (notest_runtimefixture), so my previous fix on the fixture didn't cover it. Same pattern in:test_get_protected_runtime_falls_back_to_get_runtime(inline_instance = NullRunRuntime(...))_make_test_runtime/_make_runtime_with_mocked_authhelpers intest_runtime_branches.py_test_runtimefixture intest_toolbox_langgraph.pyWhat changed
New
make_test_runtimefactory fixture intests/conftest.py— pinsNULLRUN_WAL_PATHtotmp_path, stubs_do_flush/_do_flush_locked/_client, resets the singleton around the factory. This is the canonical helper that all inline-runtime tests should use.4 inline
NullRunRuntime(...)calls intest_protect_branches.pyreplaced withmake_test_runtime():test_protect_async_kill_re_raises_WorkflowKilledInterrupttest_get_protected_runtime_falls_back_to_get_runtimetest_runtimefixture which already hadtmp_pathfrom fix(tests): pin test_runtime WAL to tmp_path to avoid cross-Python flake #62)test_runtime_branches.py: local_make_test_runtime/_make_runtime_with_mocked_authhelpers now setNULLRUN_WAL_PATHto a per-calltempfile.mkdtemp()before constructing the runtime.setdefaultso an outer-session pinning (frommake_test_runtime) is preserved.test_toolbox_langgraph.py: autouse_test_runtimefixture now takestmp_pathand pinsNULLRUN_WAL_PATH.Verified locally
pytest tests/ -n auto(Python 3.11):test_actions.py::TestPauseAction::test_is_paused_respects_cooldown— pre-existing flake on master (verified bygit stash+ repro on bare master, NOT introduced by this commit).ruff check src/: All checks passedmypy src/: Success: no issues found in 34 source filesPublic API
Unchanged. No
SDK_MIN_VERSIONbump. Backends on 1.0.0 keep working unchanged. No version bump needed.