Goal

The test results here will be the subject of an upcoming AWS blog post. However, I need a place to publish all of the raw data, which is quite long. A GitHub issue on moby seems like one obvious choice.

Post: https://aws.amazon.com/blogs/containers/preventing-log-loss-with-non-blocking-mode-in-the-awslogs-container-log-driver/

Users can use this data to decide if they want to switch to non-blocking mode and what max-buffer-size do they want to configure.

Background

• blocking (default): When logs can not be immediately sent to Amazon CloudWatch, calls from container code to write to stdout/stderr will block/freeze. The logging thread in the application will block/halt, which may prevent the application from functioning and lead to health check failures and task termination. Container startup will fail if the required log group or log stream can not be created.
• non-blocking: When logs can not be sent to Amazon CloudWatch, they are stored in an in-memory buffer configured with the max-buffer-size option. When the buffer fills up, logs are lost. Calls to write to stdout/stderr from container code will not block and will return immediately. With Amazon ECS on Amazon EC2, container startup will not fail if the required log group or log stream can not be created. With Amazon ECS on AWS Fargate, container startup always fails if the log group or log stream can not be created regardless of the mode configured.

The default blocking mode can compromise application availability. When logs can not be uploaded to CloudWatch, back-pressure in the log driver can freeze the application. Please read the AWS container blog Choosing container logging options to avoid backpressure for more information and a guide for testing your application under a logging back-pressure scenario.

The concern with non-blocking mode is the risk of log loss if/when the buffer fills up. If the CloudWatch API is unavailable, log loss is expected; however, in the happy case where CloudWatch PutLogEvents calls can be made, is non-blocking mode safe for applications that log at a high rate? Is the default 1MB max-buffer-size sufficient to prevent log loss or would most users need to increase it? We performed tests to quantify this risk.

What did the tests find?

Please be aware that the results discussed in this issue do not represent a guarantee of performance. We are simply sharing the results of tests that we ran.

Here are the key findings:

• max-buffer-size of 4+ MB does not show any log loss for <= 2MB/s log output rate from the container
• max-buffer-size of 25+ MB does not show any log loss for <= 5MB/s log output rate from the container
• Above 6 MB/s, the performance of the AWSLogs driver is less predictable/consistent. For example, there was an outlier test failure with a 100 MB buffer and 7 MB/s.
• The max-buffer-size setting controls the byte size of messages in a go slice. It does not directly constrain the memory usage, because Go is a garbage collected language. One test suite noted the real size of the queue on average is often fairly small, generally less than 500KB. Memory usage will climb to the limit occasionally during periods of latency or increased log throughput.
• AWSLogs driver can upload consistently at a much higher rate when sending logs to the CloudWatch API in the same region as the test task, due to lower latency connection to CloudWatch. Cross-region log upload is less reliable; moreover, it violates the best practice of region isolation. Cross-region log push also incurs higher network
• For cross-region log push, a max-buffer-size of 50 MB is recommended to ensure there is low risk of log loss.
• The results are similar for Amazon ECS on EC2 launch type compared with the Fargate launch type.
• There is no visibility into logs lost by the non-blocking mode buffer. There is no log statement or metric emitted by the Docker Daemon when loss occurs. Please see the proposal on Github: Proposal: Metrics for log driver log loss, logs sent, throughput, in Docker Stats or prometheus metrics #45953

How can I find my log throughput?

Use the IncomingBytes metric in CloudWatch Metrics to track the ingestion rate to your log group(s). Assuming that all containers send at roughly the same rate, you can then divide the log group ingestion rate by the number of containers. Then you have the rate for each individual container. It is recommended to over-estimate the log throughput from each container; log output may spike occasionally, especially during incidents. If possible, calculate your throughput during a load test or recent incident. It is recommended to use the peak log output rate over a time interval of a minute or less, to account for bursts in throughput.

How were tests run?

The code used for benchmarking can be found on GitHub. EC2 tests were performed on Docker version v20.10.25. Fargate tests were performed on platform version 1.4.

Each log loss test run was an Amazon ECS Task that sends 1 GB of log data to CloudWatch Logs with the AWSLogs Driver. It then queries CloudWatch Logs to get back all log events and checks how many were received. Each log message has a unique ID that is a predictable sequence number.

Several thousand test runs were executed, to get statistically meaningful data on log loss.

Full Test Results

Understanding the Data Tables

For each set of dimensions (the test run data aggregated across some set of values, like log size), there are 3 data tables.

For all tables, the horizontal axis is increasing values of max-buffer-size, the vertical axis is increasing values for log output rate from the container.

Summary Data Tables

This table type shows how many test runs were successful out of the total set of test runs for the given throughput and max-buffer-size. Successful means that no logs were lost in the entire test run. This is the strictest way of viewing the data, and also the most relevant- customers expect zero log loss.

The table uses emoji symbols to make it easier to view the data at a glance. They mean:

• ✅ => no logs were lost in any of the test runs. All were successful.
• ❕ => grey exclamation mark means only a single test run showed log loss. This case was almost completely successful. (And at least 90% of the test runs were successful. There are cases where only a few test task were run for a single combination of throughput and buffer size. The 90% test guards against a case where for example 2/3 runs were successful, which is actually 33% failure and thus should not be given a “green” positive emoji).
• ❗️ => red exclamation mark means that 95+% of runs were successful.
• ❌ => red “X” mark means that 90+% of runs were successful.
• 🚨 => indicates very non-trivial loss and that less than 90% of runs were successful (more than 10% of runs had log loss).

The number before the emoji is {successful runs}/{total runs}. If there was at least one failed test run, number after the emoji is the percent of logs lost in the worst failed test run.

Average Log Loss Data Tables

This table type shows the average % of logs lost across all test runs for the same throughput and max-buffer-size. This is also the same as the % of logs lost if all test runs were a single longer running test.

The table uses emoji symbols to make it easier to view the data at a glance. They mean:

• ✅ => no logs were lost in any of the test runs. All were successful.
• ❕ => grey exclamation mark means average log loss was 0.5% or less. 99.5% or more of logs were sent successfully.
• ❗️ => red exclamation mark means average log loss was 1% or less. 99% or more of logs were sent successfully.
• ❌ => red" "X" mark means average log loss was 5% or less. 95% or more of logs were sent successfully.
• 🚨 => indicates very non-trivial loss and log loss was greater than 5%.

Standard Deviation Log Loss Data Tables

This table shows the standard deviation in % log loss across test runs. This table shows how variable/random/unpredictable the AWSLogs driver is under different throughputs and buffer sizes in non-blocking mode.

The table uses emoji symbols to make it easier to view the data at a glance. They mean:

• ✅ => Standard deviation in log loss was less than 1%.
• ❕ => grey exclamation mark means standard deviation in log loss was less than 2%.
• ❗️ => red exclamation mark means standard deviation in log loss was less than 5%.
• ❌ => red “X” mark means standard deviation in log loss was less than 10%.
• 🚨 => indicates very non-trivial variation between test runs of 10% or more.

Log loss test results: in-region

These are the test results for tasks sending to CloudWatch in the same region as the tasks.

Summary of All Test Runs

These are the results only aggregated by max-buffer-size and log throughput. The results are not broken down by any additional dimensions like log message size or launch type.

Summary Data Table

Average Log Loss Data Table

Standard Deviation Log Loss Data Table

Much more data is in the comments, I ran into single message length limit on GitHub