Cherry-pick of PR #27242 to 7.x branch. Original message:

What does this PR do?

This PR accomplishes a few different things:

• Adds support for cgroupsV2
• Continues our deprecation of elastic/gosigar, by moving the cgroups code into libbeat. A considerable portion of the code here is pre-existing code from gosigar that's been reorganized.
• Moves the opt library from metricbeat/internal/metrics to libbeat, since we use it here too.
• Refactors the existing cgroups V1 code to be more in line with the refactor of the system module, removing the MapStr manipulation code in favor of having the data format hard-coded into the structs, and makes the cgroups code overall more "metrics first"

Note that supporting cgroups V1 and V2 involves some compromises, as the two versions structure themselves and report data fairly differently, and as such, most of the metrics APIs require the consumer to differentiate between V1 and V2 if they want to access raw metrics. V1 and V2 can also cooexist on the same system, so this must happen on a pid-by-pid basis.

Also, I'm still testing this on V1 and mixed v1/v2 systems, but the code is otherwise ready for review.

There's also the issue of dashboards. The system Docker dashboard relies on many of the V1 fields that aren't present in V2, and I'm not really sure how to deal with them. Last I tried, we don't really have any mechanism for dashboards to operate with the logic of "use this field if present, otherwise use this other field." We may also just want to re-write the dashboard entirely to just use fields that are present in V1 and V2, at the expense of losing some of the visualizations.

Also keep in mind that most of system/process is going to be aggressively refactored after this, so any issues, unless they're serious, with the code in libbeat/metric/system/process and metricbeat/system/process will almost certainly be fixed as part of 7.16.

What's up with cgroupsV2

Cgroups V2 introduces a few major changes compared to cgroups V1, which necessitated a lot of extra code:

• A new hierarchy for processes. This "unified" hierarchy requires new logic for code that wants to track processes in a cgroup
• New controllers. The various resource controllers (cpuacct, memory, etc) have changed dramatically, and in a few cases have merged with other controllers for V2. This requires entirely new metrics code.

File structure

For ease of browsing through this PR (Github honestly isn't great at presenting large PRs), here's the breakdown of what's in here:

libbeat/metric/system/cgroup // The entirety of the cgroup implementation. The files in this base directory are (mostly) unchanged files from gosigar.
├── cgcommon // Files that are shared by cgv1 and cgv2. Mostly struct definitions and helper code
├── cgv1 // Metrics  for cgroups v1. This is almost entirely code from elastic/gosigar that's been refactored.
├── cgv2 // metrics for cgroups v2. This is entirely new code.
└── testdata // data used by by the various *_test.go files

This also includes:

• libbeat/opt Which is mostly existing code that was moved from metricbeat/internal.
• Changes to cmd/instance, add_process_metadata and add_docker_metadata to migrate away from gosigar
• changes to metricbeat/internal to deal with moving the libbeat/opt code.

Why is it important?

Cgroups V2 is coming, and is already default on Fedora. It's supported by most other software at this point.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

• Pull down and build
• Note what cgroups version your host is using. You can see with grep cgroup /proc/self/mountinfo
• Test the system/process metricset, as well as add_process_metadata and add_docker_metadata