Skip to content

Fix physical memory monitoring in constrained Linux docker containers#23944

Merged
bot-gradle merged 4 commits into
masterfrom
gh/pr/17628
Feb 21, 2023
Merged

Fix physical memory monitoring in constrained Linux docker containers#23944
bot-gradle merged 4 commits into
masterfrom
gh/pr/17628

Conversation

@ghale

@ghale ghale commented Feb 17, 2023

Copy link
Copy Markdown
Member

Supersedes #17628

Thrillpool and others added 3 commits February 17, 2023 10:15
@ghale ghale added the a:bug This doesn't work as expected label Feb 17, 2023
@ghale ghale added this to the 8.1 RC1 milestone Feb 17, 2023
@ghale ghale self-assigned this Feb 17, 2023
@ghale

ghale commented Feb 17, 2023

Copy link
Copy Markdown
Member Author

@bot-gradle test this

@bot-gradle

Copy link
Copy Markdown
Collaborator

I've triggered the following builds for you:

@ghale

ghale commented Feb 17, 2023

Copy link
Copy Markdown
Member Author

@Thrillpool I'd like to add you as a contributor in the release notes for 8.1. We typically use author's actual names for this with a link to their Github profile (for example, see the opening section of https://docs.gradle.org/current/release-notes.html). However, your Github profile doesn't list your name. Would you like to provide it or would you rather just use your Github handle?


public class CGroupMemoryInfo implements OsMemoryInfo {
private static final String CGROUP_MEM_USAGE_FILE = "/sys/fs/cgroup/memory/memory.usage_in_bytes";
private static final String CGROUP_MEM_TOTAL_FILE = "/sys/fs/cgroup/memory/memory.limit_in_bytes";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been working on my own version of this, and there is a few of problems with this approach:

  1. This only works for cgroups v1. In cgroups v2, there is a "unified hierarchy" where it lives at /sys/fs/cgroup/memory.max/memory.current.
  2. When there is no limit, the number in memory.limit_in_bytes (cgroups v1) is a huge number 9223372036854771712. For memory.max (cgroups v2) it is the string max. In this case, you can't use these values - you need to read from /proc/meminfo to get the total actual system memory.
  3. Not sure how much this appears in practice, but it's possible to mount the cgroup filesystems in a different directory to /sys/fs/cgroup. The best way to handle this would be:
    i. Read /etc/mtab and look for cgroup2 or cgroup .... memory entries
    ii. If these exist, use them
    iii. If they don't exist, try the default /sys/fs/cgroup path

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can make a PR with the code I have (although might have to wait until Monday, as it's getting late here in London).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the insight here. 1 and 3 are good points, but also items I think that could be addressed as follow up work (i.e. we could deliver cgroup1 support at the default location in this PR and then add support for cgroup2 later). I think item 2 is not an issue because when the cgroup1 value is a huge number, we always select the smaller of the cgroup value or the meminfo value. Assuming we also support cgroups v2 in the future, when it's "max" we would just drop out and not consider the cgroup value (i.e. non-numeric value), defaulting to meminfo.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More than happy for this to be merged in as is, it should fix our issues in CI at least. My code unfortunately targets Java 8, rather than 6, so can't immediately get it working when pasted into this repo, but here it is for posterity.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ghale is there by any chance any ticket for supporting cgroup2 also?

import java.io.IOException;
import java.nio.charset.Charset;

public class CGroupMemoryInfo implements OsMemoryInfo {

@CRogers CRogers Feb 17, 2023

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it also be possible to backport this to Gradle 7? We've started migrating to Gradle 8, but we have over 100 plugins written over 10 years and 1000s of repos, so it takes us a good long while. Without the backport I'm going to have continue to build my plugin that swaps out the OsMemoryInfo service using reflection as a stopgap for Gradle 7, which would be a real pain (we're acutely feeling the pain of this issue atm).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's definitely possible. I'll see what we can do.

@WhosNickDoglio WhosNickDoglio Jul 17, 2023

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to resurrect an old thread but was this ever backported? I can't find it in the release notes for 7.6.2 but just want to confirm.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this got lost in the noise and didn't get included in 7.6.2. I've made sure it's on the list to be included in 7.6.3.

@Thrillpool

Copy link
Copy Markdown
Contributor

@Thrillpool I'd like to add you as a contributor in the release notes for 8.1. We typically use author's actual names for this with a link to their Github profile (for example, see the opening section of https://docs.gradle.org/current/release-notes.html). However, your Github profile doesn't list your name. Would you like to provide it or would you rather just use your Github handle?

That's very nice of you! I don't like to have my name on the internet so please list me as Thrillpool.

@ghale ghale marked this pull request as ready for review February 21, 2023 12:46
@ghale ghale requested a review from eskatos February 21, 2023 12:49

@eskatos eskatos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

\o/

@ghale ghale requested a review from hythloda as a code owner February 21, 2023 14:40
@ghale

ghale commented Feb 21, 2023

Copy link
Copy Markdown
Member Author

@bot-gradle test and merge

@bot-gradle

Copy link
Copy Markdown
Collaborator

I've triggered a build for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a:bug This doesn't work as expected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants