fix Duplicated messages are sent to dead letter topic #6960 #7021
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jiazhai
approved these changes
May 23, 2020
codelipenghui
approved these changes
May 25, 2020
sijie
approved these changes
May 26, 2020
Contributor
|
/pulsarbot run-failure-checks |
zeo1995
pushed a commit
to zeo1995/pulsar
that referenced
this pull request
Jun 5, 2020
…te-update * 'website-update' of github.com:zeo1995/pulsar: (432 commits) Fixed ordering issue in KeyShared dispatcher when adding consumer (apache#7106) Fix Duplicated messages are sent to dead letter topic apache#6960 (apache#7021) [Issue 2793][Doc]--Update the TLS hostname verification for CPP and Python clients (apache#7162) [Doc]--set netty mex frame size (apache#7174) [Doc] Update for the maximum message size (apache#7171) Fixed KeyShared consumers getting stuck on delivery (apache#7105) [apache#6003][pulsar-functions] Possibility to add builtin Functions (apache#6895) [Issue 6921][pulsar-broker-common] Replaced "Paths.get(...).getParent()", because it's system dependent and uses '\' as path separator on Windows (apache#6992) Improve broker unit test CI (apache#7173) Fix typo in exception message (apache#7027) Support KeyValue Schema Use Null Key And Null Value (apache#7139) [Doc]--Update documents for support consumer priority level in failover mode (apache#7136) Add schema config to cpp and cgo docs. (apache#7137) [Doc]--Update for the maximum message size (apache#7160) [C++] Expose ZSTD and Snappy compression to C API (apache#7014) [pulsar-proxy] add proxyLogLevel into config file (apache#6948) Add multi-hosts example for bookkeeperMetadataServiceUri (apache#6998) support for termination of partitioned topic (apache#6126) Use pure-java Air-Compressor instead of JNI based libraries (apache#5390) [Issues 5709]remove the namespace checking (apache#5716) ... # Conflicts: # site2/website/scripts/split-swagger-by-version.js
cdbartholomew
pushed a commit
to kafkaesque-io/pulsar
that referenced
this pull request
Jul 24, 2020
…ache#7021) Fixes apache#6960 ### Motivation bug fix ### Modifications "UnAckedMessageTracker" will call "consumerBase.redeliverUnacknowledgedMessages". There will be 2 steps here: 1) Filter out the messages that need to enter the DLQ 2) The remaining messages will be re-delivered via RedeliverUnacknowledgedMessages request The problem appeared in the second step, when all messages were filtered out in the first step. If the MessageIdsList received by the broker is empty, it will trigger the reposting of all unackedMessages under this consumer. Other consumers will consume messages that exceed maxRedeliveryCount again, and then the messages will repeatedly enter DLQ Therefore, if all messages have been filtered out, the RedeliverUnacknowledgedMessages request should not be initiated. Just let "processPossibleToDLQ" send them to DLQ and ack. ### Verifying this change unit test: org.apache.pulsar.client.api.DeadLetterTopicTest#testDuplicatedMessageSendToDeadLetterTopic
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
…ache#7021) Fixes apache#6960 ### Motivation bug fix ### Modifications "UnAckedMessageTracker" will call "consumerBase.redeliverUnacknowledgedMessages". There will be 2 steps here: 1) Filter out the messages that need to enter the DLQ 2) The remaining messages will be re-delivered via RedeliverUnacknowledgedMessages request The problem appeared in the second step, when all messages were filtered out in the first step. If the MessageIdsList received by the broker is empty, it will trigger the reposting of all unackedMessages under this consumer. Other consumers will consume messages that exceed maxRedeliveryCount again, and then the messages will repeatedly enter DLQ Therefore, if all messages have been filtered out, the RedeliverUnacknowledgedMessages request should not be initiated. Just let "processPossibleToDLQ" send them to DLQ and ack. ### Verifying this change unit test: org.apache.pulsar.client.api.DeadLetterTopicTest#testDuplicatedMessageSendToDeadLetterTopic
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #6960
Motivation
bug fix
Modifications
"UnAckedMessageTracker" will call "consumerBase.redeliverUnacknowledgedMessages". There will be 2 steps here:
The problem appeared in the second step, when all messages were filtered out in the first step. If the MessageIdsList received by the broker is empty, it will trigger the reposting of all unackedMessages under this consumer.
Other consumers will consume messages that exceed maxRedeliveryCount again, and then the messages will repeatedly enter DLQ
Therefore, if all messages have been filtered out, the RedeliverUnacknowledgedMessages request should not be initiated. Just let "processPossibleToDLQ" send them to DLQ and ack.
Verifying this change
unit test:
org.apache.pulsar.client.api.DeadLetterTopicTest#testDuplicatedMessageSendToDeadLetterTopic
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation