Motivation

Background

How does deduplication work?

• Producer sends messages and records sequence-ids that are pending responses in memory, the variable is {pendingMessages}
• Brokers will respond to an invalid position -1:-1 if the sequence ID published is lower than the previous messages.
• Producer checks whether the next sequence id in {pendingMessages} is larger than the one that was rejected.
  • {next} > {rejected}: ignore the error, and continue work
  • {next} < {rejected}: close channels and reconnect.

Conditions that issue happened

• Replicators send all messages on a topic and do not matter what message was sent by which original producer.
  • (Highlight) it mixed messages into {pendingMessages} with message.sequenceId but ignores message.original-producer-name, which may cause the sequence-ids in {pendingMessages} is not increasing
• The producer checking the sequence ID after receiving a -1:-1 send response will fail.

Issue-1: loss messages

• Publishing in the source cluster
  • Producer-1 send 2 messages: M1(seq: 0), M2(seq: 1)
  • Producer-2 send 2 messages: M3(seq: 0), M4(seq: 1)
• Replicate messages to the remote cluster
  • Copies M1 and M2
    • {pendingMessages}: [0,1]
  • Repeatedly copies M1 and M2. and copies M3 and M4.
    • {pendingMessages}: [0,1,0,1]
    • After repeatedly copying M1 and M2, the network broke.
    • Receive send receipt:
      • seq 0, position 0:0
      • seq 1, position 0:1
      • seq 0, position -1: -1
      • seq 1, position -1:-1
    • {pendingMessages}: [empty]
• After a topic unloading.
  • The replicator will start after the topic is loaded up(backlog is 0 now).
  • The client will create a new connection.
• Result:
  • Disabled deduplication: copied [M1, M2, M1, M2]
  • Enabled deduplication: copied [M1, M2]

You can reproduce the issue by the test testDeduplicationNotLostMessage

Issue-2: frequently fails

• Original-producer-2: sent msg 3:0 with sequence-id 10
• Original-producer-1: sent msg 3:1 with sequence-id 1
• Original-producer-1: sent msg 3:2 with sequence-id 2
  -0 Replicator copies messages
  • {pendingMessages}: [10,1, 2]
• Sent 3:0 successfully
  • {pendingMessages}: [1,2]
• Resend 3:0(a duplicated publishing)
  • Receive -1:-1(new position relates to the latest publishing) for the latest send-response.
  • failed-sequenced:10 > pendingMessages[0].sequenceId: 1
  • Replicator-producer reconnects and resets the cursor, which leads to more duplicated publishing.
  • Loop to the step 1.

No test for reproducing this issue.

Modifications

Solution: replicators use a specified sequence ID(ledegrId:entryId of the original topic) instead of using the original producers’

• Original-producer-2: sent msg 3:0 with sequence-id 10
• Original-producer-1: sent msg 3:1 with sequence-id 1
• Original-producer-1: sent msg 3:2 with sequence-id 2
• Replicator copies messages with specified sequence ID(3:0)
  • {pendingMessages}: 3:0, 3:1, 3:2]
• Sent 3:0 successfully
  • {pendingMessages}: [3:1, 3:2]
• Resend 3:0(a duplicated publishing)
• Receive -1:-1(new position relates to the latest publishing) for the latest send-response.
• Ignored the error since failed-sequenced(3:0) < pendingMessages[0].sequenceId(3:2)

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x