feat(cluster)!: let topics require durable acks and flatten config by spetz · Pull Request #4092 · apache/iggy

spetz · 2026-09-08T09:03:08Z

Producer acknowledgments did not wait for recoverable storage. The
enforce_fsync option synchronized segment flushes, but replicas sent
PrepareOk before prepares reached stable storage. Consumer-offset
synchronization was one server-wide key.

This PR adds two create-only topic options, durability and
consumer_offset_durability. Each is replicated (the default) or
persisted, and neither inherits the other. Both policies write to
disk and complete after VSR quorum commit. Persisted also requires
recoverable stable-storage copies on the quorum, or a local sync in a
single-replica group.

In replicated groups, a persisted policy adds a bounded per-partition
prepare WAL, sized by partition.wal_bytes_max. WAL records reference
message bodies in segment files, and hard links keep them until
reclamation. Replicas forward each prepare before their own WAL write
completes. Reclamation waits for durable materialized state, and
missing history or storage errors fail closed.

AWS i4i benchmarks found two costs outside the WAL, which this PR
removes. Produce admission zero-filled each request buffer before the
copy, and repair ran for operations that were already resident.

The configuration drops [system]. Its path key and remaining
tables move to the root, and IGGY_SYSTEM_* variables lose SYSTEM_.
This PR removes the stream, topic and partition path keys,
archive_expired, recreate_missing_state and
consumer_offset_enforce_fsync. The server refuses to boot with a
relocated key or a stored topic with enforce_fsync=true.

The Rust, Java, C#, Go, Node, Python, PHP and C++ SDKs, the CLI and
the benchmark expose both options. The HTTP Iggy-Durability header
reports replicated or persisted instead of replicated-memory.
Poll auto-commit and ack=none produce stay asynchronous.

A deterministic simulator storage model injects crashes, power loss
and torn writes into WAL tests. Cluster tests cover crashes and
corruption, and a compatibility test boots a baseline data directory.
The in-memory partition simulator does not run persisted topics.

codecov · 2026-09-08T10:25:42Z

Codecov Report

❌ Patch coverage is 90.90754% with 532 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.39%. Comparing base (d75890f) to head (8e1ed1d).

Files with missing lines	Patch %	Lines
core/partitions/src/state_transfer.rs	67.06%	92 Missing and 17 partials ⚠️
core/journal/src/partition_journal/segments.rs	88.07%	33 Missing and 48 partials ⚠️
core/partitions/src/persistence.rs	93.67%	49 Missing and 19 partials ⚠️
core/simulator/src/storage.rs	85.20%	17 Missing and 41 partials ⚠️
core/journal/src/durable_storage.rs	81.81%	34 Missing and 14 partials ⚠️
core/server/src/partition_helpers.rs	93.25%	14 Missing and 10 partials ⚠️
core/server/src/segment_recovery.rs	92.00%	12 Missing and 4 partials ⚠️
core/common/src/types/options/mod.rs	94.70%	1 Missing and 8 partials ⚠️
core/configs/src/configs_impl/file_provider.rs	85.00%	7 Missing and 2 partials ⚠️
core/server/src/http/handlers.rs	74.28%	5 Missing and 4 partials ⚠️
... and 23 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4092      +/-   ##
============================================
+ Coverage     86.03%   86.39%   +0.35%     
- Complexity     1449     1455       +6     
============================================
  Files          1248     1258      +10     
  Lines        195841   205504    +9663     
  Branches     161138   170716    +9578     
============================================
+ Hits         168495   177540    +9045     
- Misses        23212    23501     +289     
- Partials       4134     4463     +329

Components	Coverage Δ
Rust Core	`87.33% <90.90%> (+0.37%)`	⬆️
Java SDK	`67.58% <ø> (+0.04%)`	⬆️
C# SDK	`76.97% <ø> (-0.09%)`	⬇️
Python SDK	`91.34% <ø> (+0.01%)`	⬆️
PHP SDK	`85.61% <ø> (-0.05%)`	⬇️
Node SDK	`96.26% <ø> (+0.09%)`	⬆️
Go SDK	`69.43% <ø> (+0.03%)`	⬆️

Files with missing lines	Coverage Δ
core/binary_protocol/src/primitives/options.rs	`94.35% <100.00%> (+0.02%)`	⬆️
core/cli/src/args/topic.rs	`100.00% <100.00%> (ø)`
...ore/cli/src/commands/binary_topics/create_topic.rs	`93.33% <100.00%> (+0.65%)`	⬆️
core/cli/src/main.rs	`90.24% <100.00%> (+0.06%)`	⬆️
core/common/src/traits/message_client.rs	`100.00% <ø> (ø)`
core/common/src/types/options/durability.rs	`100.00% <100.00%> (ø)`
core/configs/src/common/defaults.rs	`100.00% <100.00%> (ø)`
core/configs/src/common/displays.rs	`100.00% <ø> (ø)`
core/configs/src/common/server.rs	`100.00% <ø> (ø)`
core/configs/src/common/system.rs	`100.00% <ø> (+4.44%)`	⬆️
... and 85 more

... and 36 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

spetz · 2026-09-08T13:15:03Z

/ready

spetz · 2026-09-08T16:36:50Z

/ready

…durability

The earlier guide tied acknowledgements to flush thresholds and the removed enforce_fsync option. Document independent replicated and persisted policies, quorum WAL recovery, and asynchronous exceptions. Align related references with the implemented configuration schema. Adapted from 68dd8d38534011bcd4adddf9dd89b12cb6ea99ad. Refs apache/iggy#4092

Bounded commit walks can leave resident operations unapplied and open redundant peer repair. Check the whole requested window before opening a session, including adopted suffixes above the commit frontier. Verify that ticks drain resident backlogs while later holes still repair.

Mixed-durability topics write message bodies into both the WAL and materialized logs. Let the WAL own the shared segment inode and defer body and directory barriers until durable publication, retaining the original writers across buffered rotations. Publish completed references separately from durable progress so replicated replies keep their flush thresholds without waiting under the append lock. Preserve clean-file sync elision and recovery gates. Cover policy combinations, delayed writes, storage faults, transfers, and cluster restart after persisted offsets acknowledge predecessors.

The cleanup tests set message expiry at topic creation. On a slow cluster, the cleaner can remove segments before the pre-expiry assertions run. A fixed sleep can also end before the cleaner runs. Create the topics without expiry and enable a 100 ms expiry after setup. Poll up to 30 s for cleanup in every partition. The staggered bootstrap test waited for log lines from the late replica that announce view adoption and repair or state transfer. Normal commit processing can catch the replica up without a repair or transfer line. Read its metadata superblock view and wait for the topic partition directory instead. The consumer group scenario logged its root client in once after connect. The metadata leader can move during polling, and QUIC and WebSocket reconnects do not restore that login. Use the reconnecting root login, so the transport logs in again after each reconnect.

diegomrsantos · 2026-09-10T18:06:54Z

Thanks for explaining, @hubcio. That helps a lot. I’ll try to help with the review

Recovery rejection lacked retry and cooperative ownership coverage. Exercise real partition reads and state transfer, preserving the recovery fence, and clarify checkpoint-prepare residency.

Named writes could attest the wrong topic's durability, while WAL backpressure misclassified accepted retries. Pending ACKs could leak or force serial superblock writes, and truncation blocked the shard. Pin HTTP routing to the captured topic and check replays before WAL admission. Drain WAL ACKs after the bounded superblock pass and avoid queueing them without a WAL. Move truncation to the bounded storage worker and preserve the restart fence for waiting replacements. Keep Helm encryption compatible with released images and require an explicit layout for unknown builds. Add regression coverage for routing, replay outcomes, ACK retries, cancellation and truncation.

diegomrsantos · 2026-09-11T07:58:50Z

Thanks for the fixes, Hubert. I'd like to help strengthen confidence in the PR's core acknowledgment guarantee. Awaited success requires quorum commit under the selected policy, and persisted additionally requires recoverable copies of the operation and its required predecessors on the quorum.

The existing storage tests already exercise the predecessor dependency. I'd like to extend that coverage to the client reply through the partition and consensus code, starting with durability=replicated and consumer_offset_durability=persisted. The test would delay synchronization of preceding message bodies so a durable quorum cannot form, verify that the offset request receives no success, then allow enough replicas to finish and verify completion.

I'll first check how this fits the existing partition fixtures. From there, I'd look for missing cases involving persistence failures and stale completions after fencing, alongside controls for successful progress and independent policy selection. Let me know if someone is already covering this path.

hubcio · 2026-09-11T08:00:42Z

sounds good, but lets do that in next PRs later today. we plan to merge this one today.

…#4132) Adds extra tests scenarios after #4092 PR.

feat(cluster): add per-topic durability and flatten server config

9794f45

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Sep 8, 2026

Merge branch 'master' into topic_durability

c8d1385

fix linter

862cd81

hubcio requested changes Sep 8, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Sep 8, 2026

numinnex reviewed Sep 8, 2026

View reviewed changes

improvements

8e1681e

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Sep 8, 2026

fix

ed7cefa

hubcio requested changes Sep 8, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Sep 8, 2026

improvements

78d7a46

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Sep 8, 2026

spetz added 3 commits September 8, 2026 18:48

Merge branch 'master' into topic_durability

390d836

update readme

03780d5

Merge branch 'topic_durability' of github.com:apache/iggy into topic_…

194d0fd

…durability

spetz force-pushed the topic_durability branch from edcc53c to 194d0fd Compare September 8, 2026 19:31

update bench defaults

97f7b0c

hubcio added the breaking:storage On-disk data format change - skips backwards compat CI label Sep 8, 2026

This was referenced Sep 9, 2026

feat(server): generic external auth callout #4055

Draft

fix(partitions): complete polls on the owning shard #4119

Open

hubcio and others added 3 commits September 10, 2026 17:40

Merge branch 'master' into topic_durability

7343be4

hubcio changed the title ~~feat(cluster): add per-topic durability and flatten server config~~ feat(cluster)!: let topics require durable acks and flatten config Sep 10, 2026

hubcio and others added 3 commits September 10, 2026 21:21

fixes

df0d49f

test(server): cover consumer-group recovery joins

aed8b4f

Recovery rejection lacked retry and cooperative ownership coverage. Exercise real partition reads and state transfer, preserving the recovery fence, and clarify checkpoint-prepare residency.

Merge branch 'master' into topic_durability

4a38798

diegomrsantos reviewed Sep 10, 2026

View reviewed changes

Comment thread core/server/src/http/handlers.rs

krishvishal mentioned this pull request Sep 11, 2026

server: partition journal repair resurrects purged data after restart #3793

Open

hubcio and others added 2 commits September 11, 2026 08:24

Merge branch 'master' into topic_durability

8e1ed1d

hubcio approved these changes Sep 11, 2026

View reviewed changes

mmodzelewski approved these changes Sep 11, 2026

View reviewed changes

numinnex approved these changes Sep 11, 2026

View reviewed changes

hubcio merged commit e002750 into master Sep 11, 2026
183 of 185 checks passed

hubcio deleted the topic_durability branch September 11, 2026 08:05

github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Sep 11, 2026

diegomrsantos mentioned this pull request Sep 11, 2026

Purge can record completion after consumer offset directory sync fails #4128

Open

numinnex mentioned this pull request Sep 11, 2026

test(simulator): cover durability failures no existing test can reach #4132

Merged

spetz pushed a commit that referenced this pull request Sep 11, 2026

test(simulator): cover durability failures no existing test can reach (…

91c3ca5

…#4132) Adds extra tests scenarios after #4092 PR.

Uh oh!

Conversation

spetz commented Sep 8, 2026 • edited by hubcio Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spetz commented Sep 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spetz commented Sep 8, 2026

Uh oh!

diegomrsantos commented Sep 10, 2026

Uh oh!

Uh oh!

diegomrsantos commented Sep 11, 2026

Uh oh!

hubcio commented Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

spetz commented Sep 8, 2026 •

edited by hubcio

Loading

codecov Bot commented Sep 8, 2026 •

edited

Loading