Skip to content

Forced-migration test: expect what the regular topic stored, not its messages-in count - #32

Merged
merlimat merged 1 commit into
mainfrom
fix-forced-migration-flake
Nov 22, 2026
Merged

merlimat merged 1 commit into
mainfrom
fix-forced-migration-flake

Conversation

@merlimat

Copy link
Copy Markdown
Contributor

Problem

v4_and_rust_clients_across_a_forced_migration (crates/apache-pulsar/tests/integration/bridge.rs) sometimes comes up one message short: 151 of 152 on main at e6221c7 (run 45041578216), then 137 of 138 and 160 of 161 on #30 (run 45042555249, run 45047459048, attempt 1). It fails the integration job of unrelated changes. The test expects a message that was never stored.

The test takes the regular topic's msgInCounter once the forced migration has terminated it, and expects the Rust Queue consumer to receive that many messages plus the 20 Rust ones sent after the migration. But the broker counts a message in msgInCounter when it takes the send (Producer.startPublishOperation), before the managed ledger stores it. The v4 producer (pulsar-perf, 10 msg/s) keeps sending through the migration. A send that reaches a partition after its managed ledger has entered the Terminated state is counted, then fails with ManagedLedgerTerminatedException, and the producer gets TopicTerminatedError. No consumer can receive it.

The CI logs keep only the broker's last 200 lines, which don't reach back to the migration, so the three CI failures are put down to this race by their signature, exactly one short, which the runs below reproduce. The test still expects an exact count, so a message really lost would still fail it.

Example

The window is a few milliseconds per partition, so at 10 msg/s it is hit now and then. With the v4 producer at 100 msg/s (a local change, not in this PR), the test came up exactly one short in 4 of 10 runs. In one of them, the counts for partition 1 were:

  • msgInCounter 154, while stats-internal had 153 entries (entriesAddedCounter 153, lastConfirmedEntry 33:152);
  • the Rust Queue consumer received those 153: the v4 producer's sequence ids 0 to 144 without a gap, and the last message at 33:152; and the v4 consumer read the same count (311 for both partitions, against a msgInCounter of 312);
  • the broker logged, for that partition:
    17:11:16,292 Terminating managed ledger
    17:11:16,296 Terminated managed ledger {lastConfirmedEntry=33:152}
    17:11:16,298 WARN Failed to persist msg in store {exception=Managed ledger was already terminated}
    
    and the v4 producer Received send error from server … TopicTerminatedError.

The test expected the counter:

// The regular topic is terminated: what it stored is final.
let legacy = messages_in(env, &topic, partitions);
let from_queue = text(&receive_all(&queue, legacy + 20).await);
assert_eq!(from_queue.len(), legacy + 20);

Change

The test takes what the regular topic stored from the v4 consumer. Once that consumer has read the terminated topic to its end (its subscription has no backlog on either partition), the messages it printed are exactly the stored ones, and the Rust Queue consumer must receive as many, plus the 20 Rust messages:

let counted = messages_in(env, &topic, partitions);
let drained = || attached_and_drained(env, &topic, partitions, "v4");
wait_for(drained, "the v4 consumer reads the regular topic to its end").await;
let from_v4 = env.stop(v4_consumer);
let from_queue = text(&receive_all(&queue, from_v4.len() + 20).await);
assert_eq!(from_queue.len(), from_v4.len() + 20);

The expected count stays exact, so the test still catches a single lost message. The counter keeps its other job: it must not move from the end of the Rust sends to the v4 producer's refusal, so the v4 producer stores nothing after the migration. Neither the client nor the broker changes.

Testing

  • With the v4 producer at 100 msg/s, where the old test failed 4 of 10 runs, the new one passed 30 of 30. In 10 of them the broker log was kept: the termination failed a v4 send in 6, each of which the old test would have failed.
  • At the stock 10 msg/s it passed 40 of 40 (no send hit the window locally, so this shows only that nothing else regressed), and the full integration suite passed.

…messages-in count

v4_and_rust_clients_across_a_forced_migration came up one message short
now and then (151 of 152, 137 of 138, 160 of 161 in CI). It expected
the Rust Queue consumer to receive the regular topic's msgInCounter,
read once the forced migration had terminated the topic, plus the 20
Rust messages sent after the migration.

The broker counts a message when it takes the send, before the managed
ledger stores it. The v4 producer keeps sending through the migration,
so a send that reaches a partition after its managed ledger was
terminated is counted, then fails with TopicTerminated: no consumer can
receive it. With the v4 producer at 100 msg/s the test failed 4 runs
of 10, each time with one partition's msgInCounter one above its
stored entries and a "Failed to persist msg in store" warning at the
termination, while the Rust and the v4 consumers read the same, stored,
messages.

The v4 consumer, once it has read the terminated topic to its end,
gives what the topic stored: the Rust Queue consumer must receive as
many, plus the 20 Rust messages. The count stays exact. The counter
still checks that the v4 producer stores nothing after the migration.
@merlimat
merlimat merged commit b8028b0 into main Nov 22, 2026
11 checks passed
@merlimat
merlimat deleted the fix-forced-migration-flake branch November 22, 2026 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant