Skip to content

fix(storage): preserve staged source on worker shutdown - #5888

Draft
ehz0ah wants to merge 1 commit into
volcengine:mainfrom
ehz0ah:fix/add-resource-shutdown-recovery
Draft

ehz0ah wants to merge 1 commit into
volcengine:mainfrom
ehz0ah:fix/add-resource-shutdown-recovery

Conversation

@ehz0ah

@ehz0ah ehz0ah commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Description

Preserve a staged add-resource source when a queue worker is interrupted during shutdown. QueueFS keeps the unacknowledged message for restart recovery, so the processor must not delete the input required by that replay.

The same asyncio.CancelledError is also used for explicit user cancellation. The handler now reads the durable task state and keeps cleanup only for CANCELLING or CANCELLED tasks. A missing or unreadable task snapshot defaults to preserving recovery artifacts.

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

Fixes #5887

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • Check the current add-resource task state when processing is cancelled.
  • Preserve the staged source for worker-shutdown interruption and QueueFS replay.
  • Keep staged-source cleanup for explicit user cancellation and cover both paths with regression tests.

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

Commands run:

uv run --offline --with pytest --with pytest-asyncio pytest -o addopts='' -q tests/parse/test_feishu_parser_api.py tests/service/test_resource_service_reserved_target_cleanup.py tests/service/test_task_queue_middleware.py
git diff --check
uvx --offline ruff check openviking/storage/queuefs/add_resource_processor.py tests/parse/test_feishu_parser_api.py

Result: 81 tests passed. Ruff and git diff --check passed.

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

Not applicable.

Additional Notes

The pre-fix regression failed because delete_temp was awaited once after a RUNNING task received CancelledError. The fixed regression preserves that source. The explicit CANCELLING control still deletes it.

ruff format --check reports one existing long monkeypatch.setattr line in tests/parse/test_feishu_parser_api.py; this PR does not change that line. The additions in this PR need no formatting changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

[Bug]: Worker shutdown deletes staged add-resource source needed for replay

1 participant