I am seeing an issue where tasks are becoming reserved even though the settings should prevent this (late ack, -Ofair, and prefetch of 1). This is only happening intermittently, about once a day. Not using any eta or countdown tasks.

This is causing tasks to line up behind other long running tasks, thereby causing delays. Our system has very long running tasks, so this is problematic. We are also unable to split long running and short running tasks apart, as it is not known ahead of time which one will take long.

Typing celery inspect report returns following (with application specific details removed):

 software -> celery:3.1.23 (Cipater) kombu:3.0.35 py:2.7.10
                    billiard:3.3.0.23 redis:2.10.5
        platform -> system:Linux arch:64bit imp:CPython
        loader   -> celery.loaders.app.AppLoader
        settings -> transport:redis results:redis://..../

        CELERY_ACKS_LATE: True
        CELERY_SEND_TASK_ERROR_EMAILS: True
        CELERY_DISABLE_RATE_LIMITS: True
        CELERY_ACCEPT_CONTENT: ['pickle', 'json']
        CELERY_IGNORE_RESULT: False
        CELERY_RESULT_DBURI: 'redis://....'
        CELERYD_CONCURRENCY: 8
        CELERY_ROUTES: None
        CELERYD_PREFETCH_MULTIPLIER: 1
        BROKER_URL: 'redis://.../'
        CELERY_RESULT_BACKEND: 'redis://.../'

-Ofair is also enabled on the command line.

Steps to reproduce

Unable to reproduce this consistently locally, this seems to happen on production.

I tried upgrading to celery 3.1.25 but did not help. Also tried with --without-gossip --without-heartbeat which did not help either. I am unable to test with 4.0 since this is in prod.

Expected behavior

Given the settings above, I would expect to see no reserved tasks.

Actual behavior

Tasks are becoming reserved intermittently, causing them to be delayed.