Pickle serialization kills multiple workers #5075
Description
First thing's first, the config (celery report):
broker_url: 'amqp://<username>:********@vlb:5672//'
result_backend: 'rpc://<username>:********@vlb:5672/'
include: [modules]
result_expires: 3600
worker_prefetch_multiplier: 1
worker_pool_restarts: True
broker_pool_limit: 700
accept_content: {'pickle'}
task_serializer: 'pickle'
result_serializer: 'pickle'
event_serializer: 'pickle'
When starting a celery worker on a specific amqp broker, everything starts out fine. It receives pickle-serialized tasks, returns the value and everything seems alright.
The problem arises when I start a new worker, consuming from the same broker.
Both workers nearly immediately die with a long exception (same for both):
[2018-09-26 19:42:16,028: CRITICAL/MainProcess] Unrecoverable error: ContentDisallowed('Refusing to deserialize disabled content of type pickle (application/x-python-serialize)',)
Traceback (most recent call last):
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
self.blueprint.start(self)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
return self.obj.start()
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 322, in start
blueprint.start(self)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 598, in start
c.loop(*c.loop_args())
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop
next(loop)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 354, in create_loop
cb(*cbargs)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/kombu/transport/base.py", line 236, in on_readable
reader(loop)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/kombu/transport/base.py", line 218, in _read
drain_events(timeout=0)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/amqp/connection.py", line 491, in drain_events
while not self.blocking_read(timeout):
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/amqp/connection.py", line 497, in blocking_read
return self.on_inbound_frame(frame)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/amqp/method_framing.py", line 77, in on_frame
callback(channel, msg.frame_method, msg.frame_args, msg)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/amqp/connection.py", line 501, in on_inbound_method
method_sig, payload, content,
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/amqp/abstract_channel.py", line 128, in dispatch_method
listener(*args)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/amqp/channel.py", line 1597, in _on_basic_deliver
fun(msg)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/kombu/messaging.py", line 624, in _receive_callback
return on_m(message) if on_m else self.receive(decoded, message)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/celery/worker/consumer/gossip.py", line 201, in on_message
_, event = prepare(message.payload)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/kombu/message.py", line 207, in payload
return self._decoded_cache if self._decoded_cache else self.decode()
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/kombu/message.py", line 192, in decode
self._decoded_cache = self._decode()
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/kombu/message.py", line 197, in _decode
self.content_encoding, accept=self.accept)
File "/home/ubuntu/.virtualenvs/python36/lib/python3.6/site-packages/kombu/serialization.py", line 256, in loads
raise self._for_untrusted_content(content_type, 'disabled')
kombu.exceptions.ContentDisallowed: Refusing to deserialize disabled content of type pickle (application/x-python-serialize
As you can see the accept_content field is set to "pickle" and pickle alone.
Issue rises when using celery 4.2.0 and celery 4.2.1 as installed from global pypi.
It's a problem for us since JSON serialization won't work for our purposes, we're sending binary objects over celery.
Steps to reproduce
To reproduce, simply start two workers with the same config shown above with the same broker.
Thanks!