Repository navigation
Forced-migration test: expect what the regular topic stored, not its messages-in count - #32
Merged
Merged
Conversation
…messages-in count v4_and_rust_clients_across_a_forced_migration came up one message short now and then (151 of 152, 137 of 138, 160 of 161 in CI). It expected the Rust Queue consumer to receive the regular topic's msgInCounter, read once the forced migration had terminated the topic, plus the 20 Rust messages sent after the migration. The broker counts a message when it takes the send, before the managed ledger stores it. The v4 producer keeps sending through the migration, so a send that reaches a partition after its managed ledger was terminated is counted, then fails with TopicTerminated: no consumer can receive it. With the v4 producer at 100 msg/s the test failed 4 runs of 10, each time with one partition's msgInCounter one above its stored entries and a "Failed to persist msg in store" warning at the termination, while the Rust and the v4 consumers read the same, stored, messages. The v4 consumer, once it has read the terminated topic to its end, gives what the topic stored: the Rust Queue consumer must receive as many, plus the 20 Rust messages. The count stays exact. The counter still checks that the v4 producer stores nothing after the migration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
v4_and_rust_clients_across_a_forced_migration(crates/apache-pulsar/tests/integration/bridge.rs) sometimes comes up one message short: 151 of 152 onmainat e6221c7 (run 45041578216), then 137 of 138 and 160 of 161 on #30 (run 45042555249, run 45047459048, attempt 1). It fails the integration job of unrelated changes. The test expects a message that was never stored.The test takes the regular topic's
msgInCounteronce the forced migration has terminated it, and expects the Rust Queue consumer to receive that many messages plus the 20 Rust ones sent after the migration. But the broker counts a message inmsgInCounterwhen it takes the send (Producer.startPublishOperation), before the managed ledger stores it. The v4 producer (pulsar-perf, 10 msg/s) keeps sending through the migration. A send that reaches a partition after its managed ledger has entered the Terminated state is counted, then fails withManagedLedgerTerminatedException, and the producer getsTopicTerminatedError. No consumer can receive it.The CI logs keep only the broker's last 200 lines, which don't reach back to the migration, so the three CI failures are put down to this race by their signature, exactly one short, which the runs below reproduce. The test still expects an exact count, so a message really lost would still fail it.
Example
The window is a few milliseconds per partition, so at 10 msg/s it is hit now and then. With the v4 producer at 100 msg/s (a local change, not in this PR), the test came up exactly one short in 4 of 10 runs. In one of them, the counts for partition 1 were:
msgInCounter154, whilestats-internalhad 153 entries (entriesAddedCounter153,lastConfirmedEntry33:152);msgInCounterof 312);Received send error from server … TopicTerminatedError.The test expected the counter:
Change
The test takes what the regular topic stored from the v4 consumer. Once that consumer has read the terminated topic to its end (its subscription has no backlog on either partition), the messages it printed are exactly the stored ones, and the Rust Queue consumer must receive as many, plus the 20 Rust messages:
The expected count stays exact, so the test still catches a single lost message. The counter keeps its other job: it must not move from the end of the Rust sends to the v4 producer's refusal, so the v4 producer stores nothing after the migration. Neither the client nor the broker changes.
Testing