@auvipy @thedrow would either of you guys be able to review this? And do you think this is the right approach?

I was able to reproduce this issue locally using this example repo: https://github.com/pb-dod/celery_pyamqp_memory_leak/tree/test_redis_disconnect

I was also able to verify this fix works there too.

I think this fix is somewhat critical, because it's likely making running Celery on the Redis broker unreliable for everyone who upgraded to at least v5.2.3.

@auvipy I tested this change in production today, and I'm still seeing the same behavior where workers stop responding the worker heartbeats after about an hour (the red lines are a deployment start/finish):

I have INFO level logging turned on, and I see no indication in the logs about why the workers get stuck (the last log message is often a successful task run). Workers on different queues with very different workloads get stuck too.

I'm not seeing the issue with workers getting stuck when I use RabbitMQ as a broker.

I think this PR fixes an issue around workers getting stuck after several Redis disconnections, but now I'm not sure it fixes the primary cause of celery/celery#7276 This also explains why I was getting this issue without seeing Redis connection errors in my logs.

@auvipy I added the integration test: 34e366d

It looks like it's revealing a problem. Good call on adding it!

I assumed the two connection objects here were the same:

However:

• register_with_event_loop's connection is a Transport
• _on_disconnect's connection is a Redis Connection.

The way I have it currently won't work because I'm adding Transports as keys to on_poll_start_by_connection and those will never match with Redis connection objects.

I'll need to re-think the solution for this.

What I tested in production today effectively disabled the fix from #1476 and workers were still getting stuck.

can we close this in favor of #2007?

also should we try to extract relevant integrations tests from here to add to the test suite?