-
-
Notifications
You must be signed in to change notification settings - Fork 5k
Description
We are using Django Rest Framework with MongoEngine, Redis, Celery and Kombu, and we are getting the following error in our logs:
`[2017-03-22 06:26:01,702: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer/consumer.py", line 594, in start
c.loop(*c.loop_args())
File "/usr/local/lib/python2.7/dist-packages/celery/worker/loops.py", line 88, in asynloop
next(loop)
File "/usr/local/lib/python2.7/dist-packages/kombu/async/hub.py", line 345, in create_loop
cb(*cbargs)
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/redis.py", line 1039, in on_readable
self.cycle.on_readable(fileno)
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/redis.py", line 337, in on_readable
chan.handlerstype
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/redis.py", line 667, in _receive
ret.append(self._receive_one(c))
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/redis.py", line 678, in _receive_one
response = c.parse_response()
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 2183, in parse_response
return self._execute(connection, connection.read_response)
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 2176, in _execute
return command(*args)
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 238, in read_response
response = self._buffer.readline()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 168, in readline
self._read_from_socket()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 143, in _read_from_socket
(e.args,))
ConnectionError: Error while reading from socket: ('Connection closed by server.',)
[2017-03-22 06:26:01,868: INFO/MainProcess] Connected to redis://:******************************/1
Versions used
python==2.7.12
redis==3.2.3
kombu==4.0.2
Django==1.10.3
celery==4.0.2
amqp==2.1.1
billiard==3.5.0.2
pytz==2016.7
Django==1.10.3o
We have HaProxy infront of Redis cluster and connections, everything else work without any issues. Could you please help us trouble shoot this error ?
Or guide on what to look for and where to look for , etc ?
We really appreciate your help,
Thanks,
Darsana