Describe the enhancement:
Currently, the shipper is acknowledging the event batch based on AcceptedCount coming from the shipper as a part of the PublishReply.
This is unsafe because AcceptedCount does not mean that the actual final output (e.g. Elasticsearch) acknowledged these events from the Shipper's queue, it just means that the shipper put it in its own queue for processing. So, the data can be lost if the shipper process dies.
To fix this we need to use PersistedIndex from PublishReply or a separate PersistedIndex endpoint to acknowledge the events.
Describe the enhancement:
Currently, the shipper is acknowledging the event batch based on
AcceptedCountcoming from the shipper as a part of thePublishReply.This is unsafe because
AcceptedCountdoes not mean that the actual final output (e.g. Elasticsearch) acknowledged these events from the Shipper's queue, it just means that the shipper put it in its own queue for processing. So, the data can be lost if the shipper process dies.To fix this we need to use
PersistedIndexfromPublishReplyor a separatePersistedIndexendpoint to acknowledge the events.