daniyalzade: you're pretty much correct

There are two leaks of GearmanJobRequest objects:

When a job is first sent, GearmanClient.establish_request_connection() is called, which adds to the request_to_rotating_connection_queue dict using the request as the key. However, the only time items are removed from that dict is inside wait_until_jobs_completed(), and only if the blocking request did not time out. If wait_until_complete is set to False, or it is set to True and the request times out, then it remains in request_to_rotating_connection_queue forever.

In addition, in the instance of GearmanClientCommandHandler, an entry is added to handle_to_request_map, mapping the job handle to the request object. These entries are only removed by _unregister_request() on complete or failure events.

The fix for the first problem is relatively easy: if wait_until_completed is False, then at the end of submit_multiple_requests, each request should be purged from request_to_rotating_connection_queue

        time_remaining = stopwatch.get_time_remaining()
        if wait_until_complete and bool(time_remaining != 0.0):
            processed_requests = self.wait_until_jobs_completed(processed_requests, poll_timeout=time_remaining)
        else:
            # Remove jobs from the rotating connection queue to avoid a leak
            for current_request in processed_requests:
                self.request_to_rotating_connection_queue.pop(current_request, None)

However the command handler object currently isn't accessible at all from the client. Perhaps calls to send_job_request should have an extra parameter which indicates whether the request should be unregistered upon reaching the CREATED state.

In any case, it seems that the "don't wait until completed" behavior of python-gearman needs to be completely reevaluated, since currently there is no care taken to ensure request objects are cleaned up in the often-used non-blocking scenario.

Memory leak in V2.0.2 #10

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions