[Bug][broker] Error deserializing message，Invalid unknonwn tag type: 7 #19460
Description
Search before asking
- I searched in the issues and found nothing similar.
Version
- pulsar-server-version: 2.9.3.0.5 based on 2.9.3
- client-producer-version: c++ 2.9.3
- client-consumer-version: go-client
Minimal reproduce step
Continuously send message, receive message and ack.
What did you expect to see?
deserializing message metadata successfully
What did you see instead?
deserializing message metadata error when PersistentMessageExpiryMonitor#expireMessages, which result in topic message can not be cleaned.
markDeletePosition is 48516, so expireMessages need to get the timestamp of 48517. In our production environment, bookie read entry 69134924:48517 successfully, but broker fail in parseMessageMetadata. entry is 3 replication. message retention time is 3 day.
However, the near position such as 48516, 48518, 48519 can read successfully, only 48517 fail.
The stack is as follow
14:12:57.694 [BookKeeperClientWorker-OrderedExecutor-12-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://like/baina/like_user_event_0501016-partition-5][cksinker_like_user_event_0501016] Error deserializing message for expiry check
java.lang.IllegalArgumentException: Invalid unknonwn tag type: 7
at org.apache.pulsar.common.api.proto.LightProtoCodec.skipUnknownField(LightProtoCodec.java:270) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5]
at org.apache.pulsar.common.api.proto.MessageMetadata.parseFrom(MessageMetadata.java:1370) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5]
at org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:435) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5]
at org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:422) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5]
at org.apache.pulsar.client.impl.MessageImpl.getEntryTimestamp(MessageImpl.java:284) ~[org.apache.pulsar-pulsar-client-original-2.9.3.0.5.jar:2.9.3.0.5]
at org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor.lambda$expireMessages$0(PersistentMessageExpiryMonitor.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.3.0.5.jar:2.9.3.0.5]
at org.apache.bookkeeper.mledger.impl.OpFindNewest.readEntryComplete(OpFindNewest.java:89) ~[org.apache.pulsar-managed-ledger-2.9.3.0.5.jar:2.9.3.0.5]
at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.lambda$asyncReadEntry0$0(EntryCacheImpl.java:222) ~[org.apache.pulsar-managed-ledger-2.9.3.0.5.jar:2.9.3.0.5]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
group stats is as follow
"group_xxx" : {
"markDeletePosition" : "69134924:48516",
"readPosition" : "79441322:45498",
"waitingReadOp" : true,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 465871946,
"cursorLedger" : 79430404,
"cursorLedgerLastEntry" : 5047,
"individuallyDeletedMessages" : "[(69134924:48518..69134924:49999],(69136302:-1..69136302:49999]
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!