Is your feature request related to a problem? Please describe.
I guess I have to start with the differences between AWS SQS and GCP Pub/Sub, here are what I think are the more significant differences between them
- SQS accepts
Ack queue message after the visibility timeout is exceeded.
- But it does not work for Pub/Sub. If the
Ack Deadline is exceeded, our Ack operation will fail.
- The upper limit of
visibility timeout of SQS is 12 hours.
- The upper limit of
Ack deadline of Pub/Sub is 10 minutes.
So here is my current situation, I'm migrating our library from the AWS SDK to go-cloud to invoke Pub/Sub. (Eventually I would like to make it works like a multi-cloud/multi-queue-service support.) (Btw, I really have to say, go-cloud is truly amazing. 😄 )
But, the design logic in all my previous services was basically Queue-message-driven. I usually poll the message first and then start processing the message. When the processing is completed, I would decide whether to Ack or Nack.
I know that though they are essentially different services, the above differences are too huge, so when processing some time-consuming requests, 10 minutes is often not enough. (Pub/Sub also not allow us to Ack after reached the deadline. 😔 )
In the end my service usually can handle that message successfully, but because that message cannot be correctly Acked, that message will be judged as a processing failure by Pub/Sub, and will eventually be automatically transferred to the dead letter topic.
Describe the solution you'd like
So I hope go-cloud can add a new method to extend Ack deadline.
func (m *pubsub.Message) ExtendAckDeadline(d time.Duration) error {...}
Describe alternatives you've considered
The alternative I guess would be
For polling:
- As soon as I poll the message, I'd need to cache it into Redis or somewhere, and like marking it is an on flying message.
(Ah, a point just came into my mind. In that case, why don't I just stop using Pub/Sub in the first place? Eventually I still have to rewrite the entire message queue mechanism.)
For handling:
- Pop out the message from that queue cache(or storage), handle it.
- if succeeded, do nothing.
- if failed, push it back into queue, but add 1 failure count.
- if reached the max retry times, deliver it to the corresponding dead letter topic.
(However, there is still a problem. If it panic, the whole thing will be ruined, so another thing may be needed to provide to prevent that request from disappearing forever after panicing.)
Additional context
Sorry, I seem to have written context in the wrong place, please refer to above.
Is your feature request related to a problem? Please describe.
I guess I have to start with the differences between AWS SQS and GCP Pub/Sub, here are what I think are the more significant differences between them
Ackqueue message after thevisibility timeoutis exceeded.Ack Deadlineis exceeded, ourAckoperation will fail.visibility timeoutof SQS is 12 hours.Ack deadlineof Pub/Sub is 10 minutes.So here is my current situation, I'm migrating our library from the AWS SDK to go-cloud to invoke Pub/Sub. (Eventually I would like to make it works like a
multi-cloud/multi-queue-servicesupport.) (Btw, I really have to say, go-cloud is truly amazing. 😄 )But, the design logic in all my previous services was basically Queue-message-driven. I usually poll the message first and then start processing the message. When the processing is completed, I would decide whether to
AckorNack.I know that though they are essentially different services, the above differences are too huge, so when processing some time-consuming requests, 10 minutes is often not enough. (Pub/Sub also not allow us to
Ackafter reached the deadline. 😔 )In the end my service usually can handle that message successfully, but because that message cannot be correctly
Acked, that message will be judged as a processing failure by Pub/Sub, and will eventually be automatically transferred to the dead letter topic.Describe the solution you'd like
So I hope
go-cloudcan add a new method to extendAck deadline.Describe alternatives you've considered
The alternative I guess would be
For polling:
(Ah, a point just came into my mind. In that case, why don't I just stop using Pub/Sub in the first place? Eventually I still have to rewrite the entire message queue mechanism.)
For handling:
(However, there is still a problem. If it
panic, the whole thing will be ruined, so another thing may be needed to provide to prevent that request from disappearing forever afterpanicing.)Additional context
Sorry, I seem to have written context in the wrong place, please refer to above.