JEP draft: Automatic Heap Sizing for ZGC

Summary

Enhance the Z Garbage Collector (ZGC) to select the size of the Java heap based on the needs of the application and the resources available from the environment. Further, ensure the JVM is a good neighbor to other processes in the system that contend for the same resources.

Goals

• Make the default heap sizing policy good out of the box.
• Dynamically adapt the heap size to changes in the workload and containing system.
• Provide users with the same application throughput as a manually and well-tuned heap size.
• Ensure existing command-line options for tuning and sizing the heap continue to work.

Non-Goals

• It is not a goal to find the optimal heap size.

Motivation

Java objects are allocated in the JVM's heap, an area of memory controlled by the JVM for managing objects. A garbage collector such as ZGC provides application developers with the illusion of an infinite heap.

ZGC is a tracing garbage collector: To reclaim memory in the heap that is no longer needed by the application, ZGC follows references from one object to another, starting from the “roots” - such as the local variables of each method on the stack and the static fields of loaded classes - and thereby finds all the reachable ("live") objects in the heap. Then, ZGC compacts the heap: It finds regions of the heap with many unreachable ("dead") objects, copies live objects out of those regions, and frees the memory used by those regions.

Accordingly, the cost of garbage collection is related to the number of live objects, not the number of dead objects, or the amount of memory used by objects, or the total size (that is, capacity) of the heap. The best way to minimize the cost of garbage collection is to let the JVM use more memory for the heap so that more objects can be allocated before garbage collection is needed at all. With less frequent garbage collection, the CPU spends more time running the application.

Configuring the maximum heap size is difficult

There is no maximum heap size that is optimal across all applications, but it would be inconvenient to force users to always specify the maximum heap size. Accordingly, the JVM uses a longstanding heuristic of 25% of physical memory as the maximum heap size by default. For example, on a machine with 128GB RAM, the JVM will use up to 32GB for the heap.

Many users, however, want or need to specify the maximum heap size – the -Xmx setting -- based on their understanding of application workloads. Choosing the right value for -Xmx is difficult even for expert users. If set too low, the application can run out of memory; if set too high, the operating system may not have enough memory to service the application, let alone other applications, and will terminate memory-intensive processes. In principle, users should find the right value by measuring the application's memory use, throughput and latency for different values of -Xmx in an experimental setup that provides a representative workload. In practice, such experiments are challenging to get right.

The -Xmx option sets the hard limit on the heap size that cannot be exceeded. The G1 and Parallel GCs have had a policy for at least using less heap memory than allowed by -Xmx, based on a configurable -XX:GCTimeRatio of wall clock time spent doing GC vs running the application. That logic never translated well to concurrent GCs like ZGC. Hence, the -Xmx option has since the first experimental release of ZGC in JDK 11 (JEP 333) been used to instruct the GC to use as close to that maximum limit as possible to minimize the CPU spent on GC work. The implication is that when using the -Xmx option with ZGC, it should be sized large enough to accommodate the largest spike in demand the application expects to see (as described in the GC tuning guide), but doing so will result in the heap growing towards this worst case.

Not sizing -Xmx for the worst case demand can result in performance regressions or even OutOfMemoryErrors if the application experiences spikes in demand that are larger than the heap can accommodate. Crafting experiments to drive the application hard enough to experience those worst case demand spikes can be difficult, resulting in experimentally derived values of -Xmx that are too low.

Even an -Xmx chosen through such a carefully crafted experiment degrades over time. The application code is updated, the application libraries are updated to new versions, and the JDK is also updated. The experiments are never re-done at the same frequency as the application is modified.

Configuring the JVM to be a good neighbor is difficult

As stated above, -Xmx acts both as a hard limit on the maximum heap size, and an explicit instruction to ZGC to minimize its CPU use by allowing the heap to grow towards that limit rather than spend CPU on GC work. This tradeoff between heap growth and CPU usage only focuses on the current JVM process. However, the memory available for the heap is affected by the memory use of other processes on the machine.

Currently, ZGC allows the user to opt into being a good neighbor by configuring a soft maximum heap size to leave more memory for other processes without risking running out of memory itself. (https://malloc.se/blog/zgc-softmaxheapsize) ZGC will strive to keep the heap below the soft maximum by performing more frequent garbage collection, but it is allowed to grow the heap beyond the soft maximum, all the way to the hard maximum (-Xmx), if not doing so would stall an allocation or throw an OutOfMemoryError. Allocation stalls are disastrous for the latency-sensitive applications that ZGC is intended to support. The soft maximum allows ZGC to leave memory for other processes while still having a reserve of memory it can use if allocation rates spike unpredictably.

For example, if the application runs well with a 2GB heap but spikes in demand mean it occasionally needs 5GB, then you could simply set -Xmx:5GB at the cost of impacting other processes. However, if you set -Xmx:5GB -XX:SoftMaxHeapSize=2G then the JVM will keep the heap under 2GB unless workload spikes occur; in that case, the JVM will expand the heap to as much as 5GB and shrink it back to 2GB once they have passed.

Unfortunately, configuring the soft maximum has similar challenges as configuring the maximum heap size. The same kinds of experiments that are necessary to determine a good -Xmx are also required to determine a good setting for -XX:SoftMaxHeapSize. A soft maximum that is too low will cause excessive CPU use by ZGC as it continually attempts to shrink the heap below the soft limit, even though there is enough memory available under the maximum heap size. A soft maximum that is too high will waste memory, just as a too high -Xmx does. Also, like any choice of -Xmx, any choice of soft maximum degrades over time if not revalidated when the application changes.

Dynamic heap sizing

Configuring -Xmx and -XX:SofMaxHeapSize correctly for an application is challenging. The JVM is better able to balance the tradeoffs between CPU and memory by dynamically adjusting the heap size. Instead of configuring -Xmx with an absolute value, users would be better served by expressing a preference between less GC activity (but more memory use) and more GC activity (but less memory use).

We propose to extend ZGC so it is aware of the overall CPU and memory pressure on the system and to allow it to act as a good neighbor to other processes, resulting in better overall system performance.

Description

We propose that ZGC:

• uses 100% of physical memory as the default maximum heap size;
• allows you to tune the cost of garbage collection by specifying how much CPU should be spent on it, as a fraction of the CPU available to the JVM;
• adopts a sensible default for its CPU use; and
• acts as a “good neighbor” by responding appropriately to changes in machine CPU and memory usage by expanding or contracting the heap, as though a soft maximum heap size is set.

Configuring ZGC in terms of CPU usage frees you from having to set the maximum or soft maximum heap sizes, and is more robust across application updates. For most applications, there is no need to tune ZGC’s CPU usage; the default will work well. You can test your application with ZGC by starting it with:

java –XX:+UseZGC …

A case study, shown later, of multiple applications running in a single container demonstrates the JVM adapting its physical memory usage and trading off CPU use to ensure that all applications run equally well within the constraints of the container.

Trading off memory and CPU

The intensity of garbage collection determines what the memory vs CPU trade-off the GC is using. More frequent garbage collection uses more CPU time, meaning less CPU time is available for the application. However, by running the garbage collector frequently, less spare heap capacity is necessary. Conversely, when the heap has lots of spare capacity, garbage collection occurs infrequently, resulting in low CPU use by ZGC and ample CPU time available for the application.

If the application receives a spike in requests, the application will consume more CPU time. To handle the requests, the application will also create more objects, shrinking the spare capacity of the heap and, traditionally, causing the garbage collector to compete with the application for some of the CPU time -- just when the application most needs it.

As discussed in JEP XXX “Faster startup and warmup with ZGC”, ZGC will expand the heap exponentially so there is a logarithmic bound to the number of expansions needed to achieve a stable heap size. This stable heap size is one that brings the GC intensity – that is, the use by the CPU by the GC as a fraction of the application’s CPU use – to meet the target GC intensity.

(( ZGC now adapts the heap size to approach a target GC intensity – represented by the amount of CPU used by the GC as a fraction of the CPU used by the application - which is a tradeoff between the amount of memory currently used for the heap (capacity) and the amount of CPU used to perform GC work. By default, ZGC chooses a target GC intensity for itself that strikes a balance between memory and CPU usage for the GC. ))

Configuring the target CPU usage of garbage collection

In JDK NN, you trade off memory use and ZGC’s CPU use. You can force ZGC to use less CPU time, meaning more CPU time for the application, at the cost of ZGC increasing the heap size which uses more physical memory. Or, you can allow ZGC to use more CPU time, meaning less CPU time for the application, if you want a lower heap memory footprint for your application.

ZGC tracks the current heap capacity: the amount of heap memory that is currently been committed, that is, has a physical RAM page assigned for it – see “A primer on modern memory management” in https://openjdk.org/jeps/8329758 for more details on memory management.

ZGC now dynamically varies an internal heap target size by default to be a good neighbor to other processes on the system and effectively tradeoff between CPU and memory use.

By default, ZGC’s target GC intensity is 12.5% of the application’s CPU use. We believe this is a reasonable balance of CPU use and memory use.

To alter the CPU usage, you configure the intensity of garbage collection via a command line option. Intensity is an integer between 1 and 10, with a default of 5. Higher values make ZGC more intensive, resulting in more frequent collections, higher CPU usage, and smaller heap sizes; lower values make ZGC less intensive, triggering less frequent collections but requiring a larger heap. For example, to use more memory in exchange for more CPU available to the application:

java –XX:UseZGC -XX:ZGCIntensity:4 –jar app.jar

The range of intensity between 1 and 10 corresponds to a range of CPU use by ZGC that is approximately 2.5% to 25%. However, this correspondence may change in future JDK releases as we improve the automatic tuning policies, which is why we use the abstract measure of intensity rather than actual CPU usage.

A good steward of CPU and memory

The heap memory currently used by live objects is continuously changing as the application creates objects and ZGC collects them. As the application warms up, ZGC rapidly expands the current heap size to stay ahead of the used heap size, as discussed in “Faster Startup and Warmup with ZGC”. Then, as the application runs, ZGC iteratively adjusts the committed heap capacity to hit the GC intensity target:

• If ZGC’s actual CPU usage rises above the target CPU usage, ZGC adjusts the heap capacity upward. Then, at the end of the next GC cycle, ZGC expands the heap so that GC activity will be less intense in the future. ZGC’s actual CPU usage will fall toward the level indicated by the GC intensity target and the performance of the application will improve.
• Conversely, if ZGC’s actual CPU usage drops below the level indicated by the GC intensity target, ZGC adjusts the heap capacity downward. At the end of the GC cycle, ZGC contracts the heap so that GC activity will be more intense in future. ZGC’s actual CPU usage will rise toward the GC intensity target and the application’s throughput will decline.

Adjustments to the heap capacity are smoothed out to avoid major swings, typically caused by transient changes in the environment. See work of Tavakolisomeh et al..

A key determinant of ZGC’s actual CPU usage is the allocation rate of the application. ZGC uses work-based sampling which tracks, indirectly, the heap occupancy ratio: the heap occupancy (size of all live objects) divided by used heap size (size of all objects). The higher the allocation rate, the more live objects; the more live objects, the higher the heap occupancy ratio and the longer a GC cycle takes. This motivates the iterative adjustment to expand the heap, giving less intense GC activity in future and therefore more CPU for the briskly allocating application.

ZGC is also a generational garbage collector, and it rebalances the sizes of the young and old generation to minimize its actual CPU use and ensure that it always adjusts towards the GC intensity target. Newly created objects are placed in the young generation of the heap while long-lived objects are promoted to the old generation; frequent minor collections are performed on the young generation while less frequent major collections are performed on the entire heap (both generations). If a series of minor collections causes ZGC’s actual CPU usage to be above the target, then ZGC infers that the young generation is undersized, and expands the heap without waiting for a major collection.

A good neighbor regarding system resources

ZGC maintains a target heap capacity for a given GC target CPU usage. However, if we let ZGC use as much memory as it wants, the system may not have enough memory available to run other processes. Therefore, in addition to monitoring the behavior of the application, ZGC also monitors the free memory on the system. If there is a decrease in free memory, ZGC will adapt its CPU usage to maintain the GC intensity and attempt to contract the heap. In response to more free memory, ZGC may expand the heap to better meet its target CPU usage.

In JDK NN, the hard maximum heap size is no longer 25% of physical memory. It is adapted to be equal to 100% of memory minus a small reserve. A reserve of memory unused by the JVM is helpful even if the JVM is the only significant process running. The unused memory acts as a safety buffer that can be used to avoid allocation stalls if the application's allocation rate rises suddenly.

If there are other processes on the system and they deplete the reserve, ZGC will contract the heap to consume less memory, at the cost of spending more CPU time on garbage collection. Both the memory use and the CPU available to the application will decrease in this case. In other words, a higher load on the system increases the intensity of garbage collection and ZGC will work to adapt the heap size and CPU use to get back to the requested intensity. Being a target intensity, there are times where ZGC will be above or below the target, always working to get back to the desired intensity.

In more detail, ZGC’s intensity is driven by the relationship of how much of the system's memory belongs to the JVM process compared to how much of the system’s CPU time is used by the JVM process. By comparing these two proportions – one spatial, one temporal – ZGC adjusts its intensity. For example, if the JVM process is using a significant fraction of the system's memory but a relatively small fraction of the CPU time, ZGC will start to collect garbage more frequently. Conversely, if the process's CPU usage is high relative to its memory consumption, ZGC will start to collect less frequently to avoid overburdening the CPU with GC tasks.

Case study in dynamic heap sizing

In this case study, a container is started that is allowed to use 16 GB of memory. In that container, four JVM processes are started. They run, respectively, a SPECjbb2015 application, a H2 database application, and two additional instances of SPECjbb2015 are then started simultanenously to show the way memory is shared between the three JVMs. The JVMs’ heap usage is monitored for ~45 minutes.

The results of the monitoring are shown below:

(( Image ))

During the initial [0, 12] minute interval, an instance of SPECjbb2015 is requesting and responding to requests at a rate of 5000 requests per second. A reasonable heap size is found that gives good performance out of the box yet fits within the container limits. The heap grows rather slowly as the CPU usage is considered rather comfortable already.

In the next [12, 24] minutes time interval, a benchmark program for the H2 in-memory database is executed. The two JVM processes share the memory allocated to the container without either exceeding the container limit. The resources are split fairly, but due to the larger memory needs of the database application, it claims a larger share of the memory. Note that the rate of change of the heap sizes is much faster this time: The database application expands its memory footprint in a burst early on and the SPECjbb2015 instance is forced to hand memory over rapidly in response. Throughout this period, the SPECjbb2015 instance is still responding to 5000 requests per second without saturating the CPU resources. Yet, the GC CPU use is inflated (not shown on this graph), leading to worse latency. After that, the database application exits.

In a subsequent phase at the [24, 30] mark, the original SPECjbb2015 application starts growing the heap to reclaim the memory it had to share with the database application. The rate of increase is rather rapid because there is a need to cool down the CPU spent unnecessarily on GC. By reducing CPU time spent, latencies can be improved upon.

In the last phase at the [30, 42] minute mark, two additional SPECjbb2015 instances are started, executing at the same 5000 requests/second request rate each. This forces the three SPECjbb2015 instances to share the resources fairly between the three programs. Within two minutes, the three instances have evenly divided the available memory ensuring good performance for each.

Alternatives

Some GCs on other platforms let users select the memory vs. CPU tradeoff by setting a target residency — the ratio between the amount of live data and total heap size. However, for the same residency, the CPU usage of garbage collecting can vary drastically, and measuring the amount of live data is challenging for generational GCs. Measuring the total CPU usage of the GC is not only more straightforward, but it more directly represents what users care about (and monitor).

Risks and Assumptions

By changing the default maximum heap size from 25% of the available memory to all available memory, there is a risk that the new heuristics use more memory than the current implementation would, and so other processes may run out of memory. However, even with a 25% default maximum heap policy there is already a risk of that happening when several JVMs using that default run on the same machine. Moreover, the dynamically updated max heap size is very likely to be able to throw OOM before exceeding the computer memory limits.