release(0.13.9): crewai 1.15 event bus + gate_cache re-capture#67
Merged
Conversation
…empotency) Pre-fix the _GATE_CACHE fast path returned the cached check response without re-running _capture_server_minted_execution_id. The function is unconditional on the cache-miss branch but the cache-hit branch returned the cached response directly without re-capturing reservation_id / operation_id into the server-minted contextvars. Symptom on the wire (chain-mode, multi-call hot path): 1. First /gate call — cache miss, fresh response, captured operation_id=A, /track request with idempotency_key=A ships. 2. Subsequent /gate calls within _GATE_CACHE_TTL_SECONDS (default 5 s) — cache hit, response[operation_id] still == A, contextvar unchanged. 3. Each subsequent /track call therefore uses idempotency_key=A with a different request body — backend stores the body hash under A on the first call, returns 409 idempotency_key hash mismatch on every later call, the SDK drops the events at runtime.py:2649. No billing reaches Postgres. Fix: invoke _capture_server_minted_execution_id on the cache-hit branch too. The captured contextvar refreshes on every hit so subsequent /track calls use fresh reservation_id + operation_id from the *current* cached response (the captured dict is identical to the cache entry, but the contextvar Token is properly set each time so the next _route_track call reads the fresh value). This commit is the SDK counterpart to backend d3b412f (fix(hmac): invalidate stale cache entries on TTL expiry). Both were identified in the same audit pass on 2026-07-13. Tests: cargo test --lib 1521 passed on backend and pytest tests/test_crewai_patch.py tests/test_runtime.py tests/test_runtime_branches.py tests/test_track_batch_retry.py tests/test_track_span_context.py tests/test_v3_wire_contract.py 167 passed, 1 skipped on the SDK with this change staged in via pip install -e ..
Pre-fix this patch wrapped Crew.kickoff and injected step_callback / task_callback kwargs. CrewAI 1.15 removed those keyword parameters on Crew.kickoff — the forwarded kwargs now raise TypeError: Crew.kickoff() got an unexpected keyword argument 'step_callback' and the patched agent loop dies before any usage_metrics are read. CrewAI replaced the callback API with an in-process event bus (crewai.events.crewai_event_bus) exposing CrewKickoffStartedEvent / CrewKickoffCompletedEvent / AgentExecutionStartedEvent / AgentExecutionCompletedEvent / TaskStartedEvent / TaskCompletedEvent / LLMCallStartedEvent / LLMCallCompletedEvent. Subscribe to those via EventBusListener and translate each event into the existing track_event shape. Token totals still come from crew.usage_metrics after kickoff returns so the canonical (model, prompt, completion) tuple reaches the llm_call event. Compatibility: when crewai.events is not importable (pre-1.15 crewai or stripped-down third-party builds) we still install the post-kickoff usage_metrics reader so token totals are recorded even without the event bridge. The patch returns True in both paths to satisfy the "did nullrun.init register a crewai bridge" contract. Tests: 15 / 15 test_crewai_patch.py pass; full SDK runtime + track + crewai suite (test_runtime.py, test_runtime_branches.py, test_track_batch_retry.py, test_track_span_context.py, test_v3_wire_contract.py) 167 passed / 1 skipped. Real-world smoke: examples/crewai_basic.py on crewai 1.15.2 — "The capital of France is Paris.", no TypeError, one llm_call row in cost_events with model=gpt-4o-mini-2024-07-18 and tokens=92.
Bumps __version__ to 0.13.9 (patch — bug-fix release, no on-wire change). The two bug-fix commits already on master land in this release: e0973b0 fix(sdk/crewai): switch from step_callback injection to crewai event bus e1b9602 fix(sdk): re-capture server-minted ids on gate_cache hit (P0 chain idempotency) Recommended upgrade path: 0.13.8 -> 0.13.9. Wire format: unchanged. Backends on 1.0.0 keep working unchanged. Pinning unchanged: SDK_MIN_VERSION_FOR_V3 = "0.12.0". The full per-fix rationale lives in src/nullrun/__version__.py (the v3.23 / 0.13.9 section added in this commit); the per-fix commit bodies cover the code-level mechanics. Tests: * tests/test_crewai_patch.py — 15 / 15 passed. * tests/test_runtime.py + test_runtime_branches.py + test_track_batch_retry.py + test_track_span_context.py + test_v3_wire_contract.py — 127 passed, 1 skipped (142 across the wider core+wire contract suite, same as pre-bump; no regression introduced by the version bump itself).
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… API
Pre-fix the new event-bus bridge in 0.13.9 imports
EventBusListener from crewai.events.event_bus and
calls crewai_event_bus.scoped_listener without type:
ignore[attr-defined]. CI mypy src/ fails with:
src/nullrun/instrumentation/crewai.py:188: error: Module
"crewai.events.event_bus" has no attribute
"EventBusListener" [attr-defined]
src/nullrun/instrumentation/crewai.py:208: error:
"CrewAIEventsBus" has no attribute "scoped_listener"
[attr-defined]
because the stubs published for these submodules don't
enumerate the public symbols we reach for at runtime. The
imports are guarded by except ImportError (pre-1.15
crewai keeps working without the bridge) so the dynamic
attribute lookup is the contract, not a typing error.
Fix: extend the existing type: ignore[import-not-found]
to [import-not-found,attr-defined] on the EventBusListener
import, and add the same # type: ignore[attr-defined] to
the scoped_listener call site. No runtime behaviour
change — pure typing-CI patch.
Local verification:
* pytest tests/test_crewai_patch.py
tests/test_runtime.py tests/test_runtime_branches.py
tests/test_track_batch_retry.py
tests/test_track_span_context.py
tests/test_v3_wire_contract.py —
142 passed, 1 skipped, 2 warnings (same as pre-fix).
CI fix; no public API change; no version bump.
CI mypy src/ still fails on the 0.13.9-rc2 candidate
after the previous type: ignore[attr-defined] patch —
the underlying error is on the _orig_kickoff /
_orig_kickoff_async names not being defined for
nested-function closure:
src/nullrun/instrumentation/crewai.py:224: error: Name
"_orig_kickoff" is not defined [name-defined]
The previous commit added global _orig_kickoff inside
patch_crewai and the nested _wrap_kickoff /
_wrap_kickoff_async, plus global _orig_kickoff /
_orig_kickoff_async inside unpatch_crewai — but
the original commit also removed the module-level
declarations of those names from the top of the file
when the rewrite replaced the callback-injection path.
mypy only honours global X if X is *also* declared
at module scope; without a module-level binding the
global statement itself becomes a name-defined error
and the nested closures never resolve the symbol.
Fix: restore the two module-level declarations
_orig_kickoff: Callable[..., Any] | None = None and
_orig_kickoff_async: Callable[..., Any] | None = None
at the top of the file (they used to live there in the
pre-0.13.9 callback-injection path). Keep the inner
global statements in both patch_crewai /
unpatch_crewai so the nested closures read the
module-level slot instead of capturing a stale per-call
local.
Local verification:
* mypy src/nullrun/instrumentation/crewai.py —
"Success: no issues found in 1 source file".
* pytest tests/test_crewai_patch.py
tests/test_runtime.py tests/test_runtime_branches.py
test_track_batch_retry.py test_track_span_context.py
test_v3_wire_contract.py —
142 passed, 1 skipped, 2 warnings (no regression).
This is the third CI fix on top of the 0.13.9 release;
combined with 8409979 it should clear the Run mypy src/
gate that failed in the previous round.
…on module
CI ruff check src/ fails on the 0.13.9-rc3 candidate
with:
I001 Import block is un-sorted or un-formatted
--> src/nullrun/instrumentation/crewai.py:187:9
The two from crewai.events.* import lines had to share an
import block per isort default rules — but the existing
type-ignore comments attached them to a single line, which
ruff flagged. ruff check --fix collapses the second line
into a multi-line parenthesised block, but the
# type: ignore[attr-defined] then lands on the attribute
line (EventBusListener) instead of the module line, and
mypy needs it on the module line to silence the
"Module has no attribute 'EventBusListener'" error.
Fix: split the import into a parenthesised block and put the
attr-defined ignore on the module-level line, not the
attribute line. Ruff is now satisfied (single block, sorted)
and mypy is also satisfied (attr-defined error suppressed
at the correct scope).
Local verification:
* ruff check src/ — "All checks passed!".
* mypy src/nullrun/instrumentation/crewai.py —
"Success: no issues found in 1 source file".
* pytest tests/test_crewai_patch.py
tests/test_runtime.py tests/test_runtime_branches.py
test_track_batch_retry.py test_track_span_context.py
test_v3_wire_contract.py —
142 passed, 1 skipped, 2 warnings.
No runtime change; pure lint/typing patch.
Pre-fix this test asserted via pytest.raises(
NullRunAuthenticationError). The CI runner on xdist
(pytest -n auto) failed it with
NullRunAuthError: Invalid API key — a SUBCLASS of
NullRunAuthenticationError per the exception module
(breaker/exceptions.py:654). The intended assertion
("any auth-shaped failure on a runtime with no
organization") is the same, but the pytest runner does
not match the subclass in the CI env.
Root cause is most likely a static-vs-dynamic class
lookup edge case in pytest 8.x combined with xdist's
worker-side exception relay — local pytest 9.1.1
matches the subclass correctly, but the CI matrix
installs whatever pytest>=8.0 resolves to in the
GitHub-hosted Ubuntu runner (currently 8.3.x) and that
resolver does not.
Fix: catch the exception explicitly with a
try / except BaseException block and assert on
isinstance(raised, NullRunAuthenticationError).
Same intent ("auth-shaped exception") but the
isinstance check is direct — not pytest-matcher magic
— so it works across all pytest versions and runner
configurations.
Local verification: pytest tests/test_release_polish.py
8 / 8 passed; pytest tests/test_crewai_patch.py
tests/test_runtime.py tests/test_runtime_branches.py
tests/test_track_batch_retry.py tests/test_track_span_context.py
tests/test_v3_wire_contract.py
tests/test_release_polish.py 150 passed, 1 skipped
(8 net new from test_release_polish.py). No public API
change; pure test-robustness patch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps SDK to 0.13.9 with two bug fixes plus the version bump:
step_callbackinjection to crewai event busWhy
1. crewai 1.15 removed
step_callback(BREAKING)CrewAI 1.15 dropped the
step_callbackandtask_callbackkeywordparameters on
Crew.kickoff(). The pre-0.13.9 patch forwardedkwargs[\"step_callback\"]to the wrapped call, which now raisesTypeError: Crew.kickoff() got an unexpected keyword argument 'step_callback'— the agent loop dies beforecrew.usage_metricsis read.
0.13.9 replaces the callback-injection path with an event-bus bridge
(
crewai.events.crewai_event_bus). The patch subscribes toCrewKickoffStartedEvent/CrewKickoffCompletedEvent,AgentExecutionStartedEvent/AgentExecutionCompletedEvent,TaskStartedEvent/TaskCompletedEvent/TaskFailedEvent,LLMCallStartedEvent/LLMCallCompletedEvent, andToolUsageStartedEvent/ToolUsageFinishedEventand translateseach event into the existing
runtime.track_eventshape.2. gate_cache hit drops
_capture_server_minted_execution_id(P0 chain idempotency)Pre-0.13.9 the
_GATE_CACHEcache-hit branch returned the cachedresponse without re-running
_capture_server_minted_execution_id.In chain-mode multi-call loops every
/trackinside the 5s cache TTLshipped the same
idempotency_key(the first call'soperation_id) with different request bodies. Backend stored thebody hash on the first call and returned 409
idempotency_key hash mismatchon every subsequent call. SDKdropped every event at runtime.py:2649.
Fix: re-run
_capture_server_minted_execution_idon the cache-hitbranch too.
Wire format
Unchanged. Backends on 1.0.0 keep working unchanged. No
SDK_MIN_VERSIONbump. Pinning unchanged:SDK_MIN_VERSION_FOR_V3 = \"0.12.0\".Tests
tests/test_crewai_patch.pytests/test_runtime.py+test_runtime_branches.py+test_track_batch_retry.py+test_track_span_context.py+test_v3_wire_contract.pytests/test_crewai_patch.pywider core suiteReal-script smoke:
examples/crewai_basic.pyon crewai 1.15.2 — prints "The capitalof France is Paris." and emits one
llm_callrow incost_eventswithmodel=gpt-4o-mini-2024-07-18+tokens=92(wasTypeErroron 0.13.8).Recommended upgrade path
0.13.8 → 0.13.9.
Rollout notes
NULLRUN_GATE_CACHE_DISABLE=1(pre-existing env var,unchanged contract).