Skip to content

pubsub: add support for extending ACK deadline #3507

@h27771420

Description

@h27771420

Is your feature request related to a problem? Please describe.

I guess I have to start with the differences between AWS SQS and GCP Pub/Sub, here are what I think are the more significant differences between them

  1. SQS accepts Ack queue message after the visibility timeout is exceeded.
  2. But it does not work for Pub/Sub. If the Ack Deadline is exceeded, our Ack operation will fail.
  3. The upper limit of visibility timeout of SQS is 12 hours.
  4. The upper limit of Ack deadline of Pub/Sub is 10 minutes.

So here is my current situation, I'm migrating our library from the AWS SDK to go-cloud to invoke Pub/Sub. (Eventually I would like to make it works like a multi-cloud/multi-queue-service support.) (Btw, I really have to say, go-cloud is truly amazing. 😄 )
But, the design logic in all my previous services was basically Queue-message-driven. I usually poll the message first and then start processing the message. When the processing is completed, I would decide whether to Ack or Nack.

I know that though they are essentially different services, the above differences are too huge, so when processing some time-consuming requests, 10 minutes is often not enough. (Pub/Sub also not allow us to Ack after reached the deadline. 😔 )
In the end my service usually can handle that message successfully, but because that message cannot be correctly Acked, that message will be judged as a processing failure by Pub/Sub, and will eventually be automatically transferred to the dead letter topic.

Describe the solution you'd like

So I hope go-cloud can add a new method to extend Ack deadline.

func (m *pubsub.Message) ExtendAckDeadline(d time.Duration) error {...}

Describe alternatives you've considered

The alternative I guess would be
For polling:

  1. As soon as I poll the message, I'd need to cache it into Redis or somewhere, and like marking it is an on flying message.
    (Ah, a point just came into my mind. In that case, why don't I just stop using Pub/Sub in the first place? Eventually I still have to rewrite the entire message queue mechanism.)

For handling:

  1. Pop out the message from that queue cache(or storage), handle it.
  2. if succeeded, do nothing.
  3. if failed, push it back into queue, but add 1 failure count.
  4. if reached the max retry times, deliver it to the corresponding dead letter topic.

(However, there is still a problem. If it panic, the whole thing will be ruined, so another thing may be needed to provide to prevent that request from disappearing forever after panicing.)

Additional context

Sorry, I seem to have written context in the wrong place, please refer to above.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions