Description

When using RabbitMQ, the confirm_publish argument to the transport layer defaults to False (this is the default of py-amqp). When RabbitMQ is stressed (hit the memory high water mark, or out of disk space) it would start dropping messages. When confirm_publish is False this results in silent message loss when invoking Celery tasks (publish side). Calls to apply_async() or delay() return immediately and without error even though the message was never published.

If BROKER_TRANSPORT_OPTIONS includes 'confirm_publish': True then the calls to to apply_async() or delay() will block until the message publish is confirmed by RabbitMQ, and if RabbitMQ remains stressed then eventually you get and exception that you can handle or would at least show in your logs.

None of this is documented. The only example given for BROKER_TRANSPORT_OPTIONS is broker_transport_options = {'visibility_timeout': 18000} # 5 hours. And that too, is nowhere near the documentation for apply_async().

Suggestions

• Provide an example of turning on confirm_publish in the documentation of BROKER_TRANSPORT_OPTIONS, with an accompanying explanation on the effects of this on RabbitMQ and publishing reliability.
• Provide equivalent documentation for other transports
• Make note on publishing reliability in the documentation of apply_async() and refer to the documentation of BROKER_TRANSPORT_OPTIONS there.