Description

Background

Ray Serve currently uses a [logging.handlers.MemoryHandler](https://github.com/ray-project/ray/blob/39b9f1734d6101835de560b0c15873d5185c1573/python/ray/serve/_private/logging_utils.py#L376C22-L376C52) to buffer logs for efficiency. This works well for high-throughput applications where logs are emitted continuously and flushed regularly due to buffer size or severity thresholds.

However, in applications with sporadic traffic, logs can remain buffered in memory for long periods, since the handler only flushes when:

• The buffer reaches capacity, or
• A record meets/exceeds flushLevel (e.g., logging.ERROR).

This means that for low-traffic workloads, logs may not appear in real time, which complicates debugging and monitoring.

Proposal

Add support for an idle timeout–based flush mechanism to the Serve logging system.

Specifically:

• Extend the existing MemoryHandler (or create a subclass) that automatically flushes its buffer if no new log records have been emitted for a configurable duration (e.g., flush_timeout_s seconds).
• The timeout should reset on each emitted record.
• When the timeout expires, the handler should call .flush() automatically.

Example Configuration

handler = TimedMemoryHandler(
    capacity=1000,
    flushLevel=logging.ERROR,
    flush_timeout_s=5.0,  # flush if idle for >5s
    target=logging.StreamHandler(sys.stdout)
)

References

• Current implementation: [logging_utils.py](https://github.com/ray-project/ray/blob/master/python/ray/serve/_private/logging_utils.py)
• Standard library reference: [logging.handlers.MemoryHandler](https://docs.python.org/3/library/logging.handlers.html#logging.handlers.MemoryHandler)

Use case

No response