[BUG] (v3.2.0) changing opensearch.cgroups.hierarchy.override causes java.lang.SecurityException exception #20522
Description
Describe the bug
By running the OS in the container, creating a sub сgroup and changing opensearch.cgroups.hierarchy.override to its path results in an exception
[2026-02-02T09:08:53,379][WARN ][o.o.m.o.OsProbe ] [test-cluster-z502-1.es-test-cluster.svc.stg-clickhouse01-z502.k8s.o3.ru] exception retrieving free physical memory
java.lang.reflect.InvocationTargetException: null
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:119) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:565) ~[?:?]
at org.opensearch.monitor.os.OsProbe.getFreePhysicalMemorySize(OsProbe.java:109) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.monitor.os.OsProbe.osStats(OsProbe.java:693) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.monitor.os.OsService.<init>(OsService.java:71) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.monitor.MonitorService.<init>(MonitorService.java:62) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.node.Node.<init>(Node.java:1028) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.node.Node.<init>(Node.java:482) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:249) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:249) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:411) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-3.2.0.jar:3.2.0]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.2.0.jar:3.2.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.2.0.jar:3.2.0]
Caused by: java.lang.SecurityException: Denied OPEN (read) access to file: /sys/fs/cgroup/common/memory.current, domain: ProtectionDomain (file:/usr/share/opensearch/lib/opensearch-cli-3.2.0.jar <no signer certificates>)
jdk.internal.loader.ClassLoaders$AppClassLoader@18b4aac2
<no principals>
java.security.Permissions@27a60d16 (
)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:347) ~[?:?]
at java.base/java.nio.file.Files.lines(Files.java:3738) ~[?:?]
at java.base/java.nio.file.Files.lines(Files.java:3829) ~[?:?]
at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:66) ~[?:?]
at java.base/jdk.internal.platform.CgroupSubsystemController.getLongValue(CgroupSubsystemController.java:125) ~[?:?]
at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getLongVal(CgroupV2Subsystem.java:58) ~[?:?]
at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getLongVal(CgroupV2Subsystem.java:65) ~[?:?]
at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getMemoryUsage(CgroupV2Subsystem.java:253) ~[?:?]
at java.base/jdk.internal.platform.CgroupMetrics.getMemoryUsage(CgroupMetrics.java:142) ~[?:?]
at jdk.management@24.0.2/com.sun.management.internal.OperatingSystemImpl.getFreeMemorySize(OperatingSystemImpl.java:235) ~[?:?]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
... 17 more
To elaborate, we are running Opensearch version 3.2.0 in a container. In order to limit process's I/O usage, we create a sub-cgroup with the desired limit and run the process within this group. The JVM picks up the cgroup information just fine, but a change to the opensearch.cgroups.hierarchy.override setting causes an exception.
As best as I can tell, this is due to hardcoded paths in the security.policy file that ignore the opensearch.cgroups.hierarchy.override configuration - https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/resources/org/opensearch/bootstrap/security.policy. The paths for systemd cgroups are specified, but systemd cannot be used in a container.
A simple workaround is to name the cgroup to match one of the allowed paths, such as user.slice. I have tested versions 2.16.0, 2.19.3, and 3.0.0 and they all seem to work fine. I'm not sure why this is, as the security.policy files for these versions appear to be similar to 3.2.0.
Related component
Other
To Reproduce
Sorry, steps are not exact
- Create cgroup named test
- Run Opensearch 3.2.0 with flag -Dopensearch.cgroups.hierarchy.override=/test in this cgroup
Expected behavior
Opensearch pickes up memory and cpu limits from cgroup
Additional Details
Host/Environment (please complete the following information):
- Base image: ubuntu:jammy
- Version 3.2.0