Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, it's very challenging to investigate issues related to message replay ("message redelivery controller"). Some examples of this include:

• The "repeated Read-and-discard when using Key_Shared mode" issue mitigated by:
  • [improve] [broker] Avoid repeated Read-and-discard when using Key_Shared mode #22245
  • [improve][ml] Filter out deleted entries before read entries from ledger. #21739
• An older mitigation: Fixed KeyShared consumers getting stuck on delivery #7105

Solution

Add topic stats and metrics for observing message replay and related Key_Shared filtering (hash blocking) behavior.

Specific Metrics to Consider

1. Number of messages in redelivery (replay)
2. For Key_Shared subscriptions: Ways to observe internal state related to blocked hashes
3. Counter for delayed delivery messages being added to delivery (replay)

Implementation Requirements

• It should be possible to detect replays in topic stats (or internal stats) and also in aggregated metrics
• The aggregated metrics should be usable in monitoring tools (e.g., Grafana dashboards)
• The specific types of metrics (counters, gauges) to be used will be determined in the detailed design phase

Expected Benefits

• Improved observability for message replay and Key_Shared behavior
• Easier troubleshooting of related issues
• Enhanced monitoring capabilities for Pulsar clusters

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!