A worker won't pause if it's busy spilling #6182
Description
Simplified code:
class WorkerMemoryManager:
def __init__(self):
pc = PeriodicCallback(self.memory_monitor, 200)
worker.periodic_callbacks["memory_monitor"] = pc
async def memory_monitor(self):
if self._memory_monitoring:
return
self._memory_monitoring = True
self._maybe_pause_or_unpause()
await self._maybe_spill()
self._memory_monitoring = FalseIf the file system is particularly slow, the spilling may take more than 200ms. This means that the worker may jump from 70% (spill threshold) to 95% (terminate threshold) while ignoring the pause threshold (80%).