Skip to content

DequeHandler prevents garbage collection of Worker instances and others #6362

@hendrikmakait

Description

@hendrikmakait

In tests that raise an exception, the DequeHandler prevents garbage collection by maintaining a reference to the raised exception and its traceback:
worker_chain
(This reference chain was created with objgraph.find_backref_chain https://mg.pov.lt/objgraph/#memory-leak-example)

Keeping those references alive prevents garbage collection of otherwise released objects and therefore testing for proper garbage collection of Scheduler, TaskState and Worker instances as requested by #6250. For example, in #6344, the failures of distributed/tests/test_worker.py::test_heartbeat_comm_closed and several other tests are caused by this.

One possible solution to enable garbage collection in tests would be to call DequeHandler.clear_all_instances() in check_instances() before checking for proper garbage collection, though there might be superior options that generally improve our logging.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions