Skip to content

test(partitions): reproduce purge offset directory sync failure - #4130

Draft
diegomrsantos wants to merge 1 commit into
apache:masterfrom
diegomrsantos:test/purge-offset-directory-sync
Draft

test(partitions): reproduce purge offset directory sync failure#4130
diegomrsantos wants to merge 1 commit into
apache:masterfrom
diegomrsantos:test/purge-offset-directory-sync

Conversation

@diegomrsantos

Copy link
Copy Markdown

Purge can record its applied generation after synchronizing a consumer offset directory fails. The offset files have disappeared from the live filesystem, but their deletion has not been confirmed durable. Recording completion at that point can prevent reconciliation from retrying cleanup after a crash.

Refs #4128.

This draft adds regression coverage based on master at e0027506f2a0379ee3bb5508427ff570e3a9a435. It contains no production fix. The existing fault injection for offset directory synchronization is shared with the purge path and counted so each failure test verifies that it reached the intended I/O error.

The fixture persists offsets for an individual consumer and a consumer group, records applied generation 4, and requests purge generation 5. It verifies that the real unlinks succeed, then hydrates the generation into a fresh partition using the production recovery method.

  • One regression injects failure when synchronizing the individual consumer directory.
  • Another injects failure when synchronizing the consumer group directory.
  • A control performs both synchronizations successfully and verifies completion.

The assertion compares (purge_succeeded, applied_generation, recovered_generation). Both failure cases expect (false, 4, 4), preserving the pending cleanup obligation. Existing behavior produces (true, 5, 5): purge returns success and advances the generation in memory and on disk despite the failed barrier. The control expects (true, 5, 5) and passes.

The two regressions intentionally remain red until the correctness fault is fixed. Failing CI is expected for this draft reproducer. It demonstrates incorrect completion after a failed durability barrier; it does not simulate actual power loss or resurrection of deleted files.

Run the focused reproduction with:

cargo test -p partitions purge_offset_directory_sync_tests -- --nocapture

Verification on macOS against the stated master commit produced one passing control and two expected failures. The full partitions library suite produced 272 passes and only the same two failures. Formatting, dependency ordering, Clippy for partitions with all features and targets, and whitespace checks pass.

Full workspace Clippy stops at existing unused_self and unnecessary_wraps findings in core/server/src/shard_allocator.rs. The Taplo script requires Bash 4.2 or newer and could not run with the available Bash 3.2. No TOML files are changed.

Purge records its applied generation even when synchronizing consumer
offset deletions fails, which can suppress cleanup during recovery.

Add deterministic failures for both offset directories and a successful
control. Verify the real unlinks and the generation recovered from disk.
The failure regressions remain red until the purge completion path is fixed.

Refs apache#4128.
@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.39%. Comparing base (e002750) to head (5373355).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4130      +/-   ##
============================================
- Coverage     86.39%   86.39%   -0.01%     
- Complexity     1455     1457       +2     
============================================
  Files          1259     1259              
  Lines        205513   205502      -11     
  Branches     170714   170703      -11     
============================================
- Hits         177550   177538      -12     
- Misses        23535    23536       +1     
  Partials       4428     4428              
Components Coverage Δ
Rust Core 87.33% <ø> (-0.01%) ⬇️
Java SDK 67.60% <ø> (+0.01%) ⬆️
C# SDK 76.98% <ø> (+0.01%) ⬆️
Python SDK 91.34% <ø> (ø)
PHP SDK 85.65% <ø> (ø)
Node SDK 96.15% <ø> (-0.09%) ⬇️
Go SDK 69.51% <ø> (+0.10%) ⬆️
Files with missing lines Coverage Δ
core/partitions/src/iggy_partition.rs 92.33% <ø> (+<0.01%) ⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant