This is a feature meta issue to support end to end acknowledgement in the shipper. Inputs must be able to be notified when their data has successfully been written to the target output to avoid data loss. End to end acknowledgement is only necessary when using the memory queue. Data published to the disk queue will be acknowledged immediately as it cannot be lost once persisted.

The delivery guarantees section of the shipper proposal describes the high level implementation of end to end acknowledgement in the shipper system. The Event Protocol section includes a preliminary definition of the acknowledgement streaming RPC that can be taken as a starting point.

The feature is considered complete when:

• A test exists to prove that data published to the shipper is asynchronously acknowledged when:
  • Data is successfully written to the output system using any retry policy.
  • Data fails to write and then succeeds when using the infinite or max retry policies.
  • Data fails to write and then is dropped when using a max retry policy.
• A test exists to prove that end to end acknowledgement behaves correctly when the shipper is restarted during publishing. Specifically, acknowledgement or queue identifiers are not reused between restarts of the process to avoid acknowledgement collisions.

The assignee of this issue is expected to create the development plan with all child issues for this feature. The following set of tasks should be included in the initial issues at a minimum:

• Add entry IDs to the memory queue beats#32541
• The disk queue must assign an auto-incrementing ID to each element #27
• Implement a mechanism to avoid acknowledgement collisions across shipper restarts. The queue ID sequence should not restart from zero when the shipper restarts for example.
• Implement the acknowledgement stream RPC to notify inputs when their messages have been acknowledged.
• Make the shipper output to acknowledge batches based on PersistedIndex not AcceptedCount beats#32329
• Beats shipper output shouldn't keep pending events in the queue #97