test(partitions): reproduce purge offset directory sync failure - #4130
Draft
diegomrsantos wants to merge 1 commit into
Draft
test(partitions): reproduce purge offset directory sync failure#4130diegomrsantos wants to merge 1 commit into
diegomrsantos wants to merge 1 commit into
Conversation
Purge records its applied generation even when synchronizing consumer offset deletions fails, which can suppress cleanup during recovery. Add deterministic failures for both offset directories and a successful control. Verify the real unlinks and the generation recovered from disk. The failure regressions remain red until the purge completion path is fixed. Refs apache#4128.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4130 +/- ##
============================================
- Coverage 86.39% 86.39% -0.01%
- Complexity 1455 1457 +2
============================================
Files 1259 1259
Lines 205513 205502 -11
Branches 170714 170703 -11
============================================
- Hits 177550 177538 -12
- Misses 23535 23536 +1
Partials 4428 4428
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purge can record its applied generation after synchronizing a consumer offset directory fails. The offset files have disappeared from the live filesystem, but their deletion has not been confirmed durable. Recording completion at that point can prevent reconciliation from retrying cleanup after a crash.
Refs #4128.
This draft adds regression coverage based on
masterate0027506f2a0379ee3bb5508427ff570e3a9a435. It contains no production fix. The existing fault injection for offset directory synchronization is shared with the purge path and counted so each failure test verifies that it reached the intended I/O error.The fixture persists offsets for an individual consumer and a consumer group, records applied generation
4, and requests purge generation5. It verifies that the real unlinks succeed, then hydrates the generation into a fresh partition using the production recovery method.The assertion compares
(purge_succeeded, applied_generation, recovered_generation). Both failure cases expect(false, 4, 4), preserving the pending cleanup obligation. Existing behavior produces(true, 5, 5): purge returns success and advances the generation in memory and on disk despite the failed barrier. The control expects(true, 5, 5)and passes.The two regressions intentionally remain red until the correctness fault is fixed. Failing CI is expected for this draft reproducer. It demonstrates incorrect completion after a failed durability barrier; it does not simulate actual power loss or resurrection of deleted files.
Run the focused reproduction with:
cargo test -p partitions purge_offset_directory_sync_tests -- --nocaptureVerification on macOS against the stated master commit produced one passing control and two expected failures. The full partitions library suite produced 272 passes and only the same two failures. Formatting, dependency ordering, Clippy for partitions with all features and targets, and whitespace checks pass.
Full workspace Clippy stops at existing
unused_selfandunnecessary_wrapsfindings incore/server/src/shard_allocator.rs. The Taplo script requires Bash 4.2 or newer and could not run with the available Bash 3.2. No TOML files are changed.