Repository navigation
Conversation
When the removed hook is followed by a hook that defines new_forward, remove_hook overwrote that hook's fn_ref.forward, which holds its new_forward, instead of fn_ref.original_forward, which points at the rest of the chain. The next hook then stopped running while still registered. With lazy group offloading plus FirstBlockCache this bypasses the cache's tail hook and later fails with a NoneType tail_block_residuals error. Rewire original_forward when the next hook has one, and add a regression test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Hi @theontho, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice. Once the PR links an issue (or gets the |
| output = self.model(input).mean().detach().cpu().item() | ||
| assert output != 0.0 | ||
|
|
||
| def test_remove_hook_keeps_next_hook_new_forward(self): |
There was a problem hiding this comment.
Could this also cover a new_forward that calls original_forward? SkipLayerHook(skip_layer=True) never follows that link, so this test cannot detect a stale link that still executes the removed inner hook.
What does this PR do?
HookRegistry.remove_hookrewires the hook registered after the removed one by assigning to itsfn_ref.forward. For a hook that definesnew_forward, though,fn_ref.forwardholds thatnew_forwardand the rest of the chain lives infn_ref.original_forward(seeregister_hook).Overwriting
forwardsilently drops the next hook'snew_forward, so that hook stops running eventhough it is still registered.
This PR rewires
original_forwardwhen the next hook has one, and keeps the previous behaviour(rewiring
forward) for hooks withoutnew_forward.How it shows up in practice
Lazy group offloading registers a temporary tracing hook and removes it after the first forward. When
FirstBlockCacheis applied on top, removing that hook bypasses the cache's tail-blocknew_forwardwhile the head block's cache hook stays active. The next step that decides to reuse the cache then
fails with:
We hit this running Wan 2.2 with group offloading plus
FirstBlockCacheConfigover a long sequence ofjobs. It reproduces on 0.42.0 and on current
main.Minimal reproduction
Tests
Added
test_remove_hook_keeps_next_hook_new_forwardtotests/hooks/test_hooks.py, which failsbefore the change and passes after it. Existing hook tests are unaffected; a chain of hooks without
new_forwardstill rewiresforwardexactly as before.Before submitting
🤖 Generated with Claude Code