Description
Recently, I worked on a project to benchmark log loss with the AWSLogs driver in non-blocking mode with different values of max-buffer-size and log output throughput. These results will be published soon and will be linked here once published.
One of the issues I see with users using the benchmark results to guide their own configurations is that there’s limited visibility into logging with log drivers. I can’t easily find the output rate of my container. When the buffer drops logs for lack of buffer space, there’s no log message or metric emitted.
Problem Statement
Log drivers have limited visibility. Ideally, users should be able to track metrics or logs on the rate of log emission from their containers and the rate of log loss vs log upload success from the driver.
With non-blocking mode, the main concern is log loss in the buffer. When logs are dropped, it is completely silent, there’s no way to track it.
The highest priority problem is lack of visibility into log loss. Tracking log throughput is secondary.
Proposal: Metrics for log driver log loss, logs sent, throughput, in Docker Stats
While the log loss visibility problem could be solved with logs emitted by the Docker daemon, this approach doesn’t scale. Users want to quantify log loss per container, and counting messages emitted by the entire daemon is inconvenient. Also, in cases of repeated high rate of log loss, the docker daemon logs would be spammed with many error statements.
The ideal place for this sort of data is to add new counters in the docker stats interface OR the prometheus metrics interface.
As shown in this diagram, log loss by the non-blocking buffer could be tracked solely by Docker, without the need for changes in any log drivers. This could make implementation simpler.
Description
Recently, I worked on a project to benchmark log loss with the AWSLogs driver in
non-blockingmode with different values ofmax-buffer-sizeand log output throughput. These results will be published soon and will be linked here once published.One of the issues I see with users using the benchmark results to guide their own configurations is that there’s limited visibility into logging with log drivers. I can’t easily find the output rate of my container. When the buffer drops logs for lack of buffer space, there’s no log message or metric emitted.
Problem Statement
Log drivers have limited visibility. Ideally, users should be able to track metrics or logs on the rate of log emission from their containers and the rate of log loss vs log upload success from the driver.
With
non-blockingmode, the main concern is log loss in the buffer. When logs are dropped, it is completely silent, there’s no way to track it.The highest priority problem is lack of visibility into log loss. Tracking log throughput is secondary.
Proposal: Metrics for log driver log loss, logs sent, throughput, in Docker Stats
While the log loss visibility problem could be solved with logs emitted by the Docker daemon, this approach doesn’t scale. Users want to quantify log loss per container, and counting messages emitted by the entire daemon is inconvenient. Also, in cases of repeated high rate of log loss, the docker daemon logs would be spammed with many error statements.
The ideal place for this sort of data is to add new counters in the
docker statsinterface OR the prometheus metrics interface.As shown in this diagram, log loss by the non-blocking buffer could be tracked solely by Docker, without the need for changes in any log drivers. This could make implementation simpler.