Skip to content

Cherry-pick #25057 to 7.x: Add cgroup.cpuacct percentages#25095

Merged
fearful-symmetry merged 3 commits intoelastic:7.xfrom
fearful-symmetry:backport_25057_7.x
Apr 19, 2021
Merged

Cherry-pick #25057 to 7.x: Add cgroup.cpuacct percentages#25095
fearful-symmetry merged 3 commits intoelastic:7.xfrom
fearful-symmetry:backport_25057_7.x

Conversation

@fearful-symmetry
Copy link
Copy Markdown
Contributor

@fearful-symmetry fearful-symmetry commented Apr 14, 2021

Cherry-pick of PR #25057 to 7.x branch. Original message:

What does this PR do?

This PR addresses #23391 and adds percentage values to the cpuacct cgroup. Although that change is fairly straightforward, I felt the need to do some refactoring to make it nice--right now, the process metricset tends to divide work between the metricset itself and libbeat in a way that doesn't seem consistent, as libbeat does almost all the heavy lifting for regular process metrics, and then for cgroup data we do all the work in the metricset.

In order to emulate how process calculates regular cpu percentages, I just had libbeat manage cgroup data the way in manages all the other process data, so we just need to add the bits that deals with cgroup logic, and everything else is handled by the existing process process code. As a result of this, most of the cgroup logic has now been moved to libbeat, since I felt it made organizational sense to not flip-flop between doing the MapStr manipulation between two different libraries.

Why is it important?

People are asking for the enhancement in #23391

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  • Pull down, build
  • Run system/process on a host with cgroups, and cgroup processes running.
  • Look for the percentages in system.process.cgroups.cpuacct, make sure the values are there, and that they make sense.

Related issues

* initial commit of cgroup cpuacct percentages

* add in percpu flag

* fix up pct function

* formatting, name cleanup

* remove old code

(cherry picked from commit b2f22fa)
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 14, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 14, 2021

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: Pull request #25095 updated

  • Reason: Aborted from #3

  • Start Time: 2021-04-14T19:44:18.241+0000

  • Duration: 198 min 38 sec

  • Commit: 6ff3b4c

Test stats 🧪

Test Results
Failed 0
Passed 3
Skipped 0
Total 3

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 36

Expand to view the steps failures

Show only the first 10 steps failures

x-pack/filebeat-Lint - make -C x-pack/filebeat check;make -C x-pack/filebeat update;make -C filebe
  • Took 2 min 1 sec . View more details on here
  • Description: make -C x-pack/filebeat check;make -C x-pack/filebeat update;make -C filebeat check;make -C filebeat update;make check-no-changes;
x-pack/filebeat-windows-2016-windows-2016 - mage build unitTest
  • Took 2 min 47 sec . View more details on here
  • Description: mage build unitTest
gsutil -m -q cp -a public-read test-build-artifacts-x-pack-filebeat-windows-2016-windows-2016-tgz gs
  • Took 0 min 3 sec . View more details on here
  • Description: @echo off gsutil -m -q cp -a public-read test-build-artifacts-x-pack-filebeat-windows-2016-windows-2016-tgz gs://beats-ci-temp/Beats/beats/PR-25095-2
x-pack/functionbeat-windows-windows-2019 - mage build unitTest
  • Took 4 min 59 sec . View more details on here
  • Description: mage build unitTest
x-pack/libbeat-Lint - make -C x-pack/libbeat check;make -C x-pack/libbeat update;make check-no-cha
  • Took 2 min 39 sec . View more details on here
  • Description: make -C x-pack/libbeat check;make -C x-pack/libbeat update;make check-no-changes;
x-pack/metricbeat-Lint - make -C x-pack/metricbeat check;make -C x-pack/metricbeat update;make -C
  • Took 5 min 27 sec . View more details on here
  • Description: make -C x-pack/metricbeat check;make -C x-pack/metricbeat update;make -C metricbeat check;make -C metricbeat update;make check-no-changes;
x-pack/metricbeat-unitTest - mage build unitTest
  • Took 2 min 35 sec . View more details on here
  • Description: mage build unitTest
x-pack/metricbeat-windows-8-windows-8 - mage build unitTest
  • Took 2 min 11 sec . View more details on here
  • Description: mage build unitTest
gsutil -m -q cp -a public-read test-build-artifacts-x-pack-metricbeat-windows-8-windows-8-tgz gs://b
  • Took 0 min 4 sec . View more details on here
  • Description: @echo off gsutil -m -q cp -a public-read test-build-artifacts-x-pack-metricbeat-windows-8-windows-8-tgz gs://beats-ci-temp/Beats/beats/PR-25095-2
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-04-14T20:34:02.053Z]   BEAT_VERSION: null
[2021-04-14T20:34:02.053Z]   BEATS: null
[2021-04-14T20:34:02.053Z]   BUILD_DIR: null
[2021-04-14T20:34:02.053Z]   COMMIT_ID: null
[2021-04-14T20:34:02.053Z]   COVERAGE_DIR: null
[2021-04-14T20:34:02.053Z]   COVERAGE_TOOL: null
[2021-04-14T20:34:02.053Z]   COVERAGE_TOOL_REPO: null
[2021-04-14T20:34:02.053Z]   DOCKER_CACHE: null
[2021-04-14T20:34:02.053Z]   DOCKER_COMPOSE_PROJECT_NAME: null
[2021-04-14T20:34:02.053Z]   DOCKER_COMPOSE: null
[2021-04-14T20:34:02.053Z]   FIND: null
[2021-04-14T20:34:02.053Z]   GOBUILD_FLAGS: null
[2021-04-14T20:34:02.053Z]   GOIMPORTS: null
[2021-04-14T20:34:02.053Z]   GOIMPORTS_REPO: null
[2021-04-14T20:34:02.053Z]   GOIMPORTS_LOCAL_PREFIX: null
[2021-04-14T20:34:02.053Z]   GOLINT: null
[2021-04-14T20:34:02.053Z]   GOLINT_REPO: null
[2021-04-14T20:34:02.053Z]   GOPACKAGES_COMMA_SEP: null
[2021-04-14T20:34:02.053Z]   GOX_FLAGS: -arch 386
[2021-04-14T20:34:02.054Z]   GOX_OS: null
[2021-04-14T20:34:02.054Z]   GOX_OSARCH: null
[2021-04-14T20:34:02.054Z]   HOME: C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c
[2021-04-14T20:34:02.054Z]   NOSETESTS_OPTIONS: null
[2021-04-14T20:34:02.054Z]   NOW: null
[2021-04-14T20:34:02.054Z]   PATH: C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\bin;C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\.gvm\versions\go1.15.9.windows.amd64\bin;C:\ProgramData\chocolatey\bin;C:\Python38;C:\Python38\Scripts;C:\tools\mingw64\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Python38\Scripts\;C:\Python38\;C:\Python27\;C:\Python27\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\ProgramData\GooGet;C:\Program Files\Google\Compute Engine\metadata_scripts;C:\Program Files\Google\Compute Engine\sysprep;C:\Program Files\infra\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\Program Files\Java\jdk1.8.0_211\bin;C:\Program Files\kubectl\bin;C:\Program Files\CMake\bin;C:\tools\mingw64\bin;C:\Program Files\dotnet\;C:\Users\jenkins\AppData\Local\Microsoft\WindowsApps
[2021-04-14T20:34:02.054Z]   PKG_BUILD_DIR: null
[2021-04-14T20:34:02.054Z]   PKG_UPLOAD_DIR: null
[2021-04-14T20:34:02.054Z]   PIP_INSTALL_PARAMS: null
[2021-04-14T20:34:02.054Z]   PROJECTS: null
[2021-04-14T20:34:02.054Z]   PROJECTS_ENV: null
[2021-04-14T20:34:02.054Z]   PYTHON_ENV: null
[2021-04-14T20:34:02.054Z]   PYTHON_ENV_EXE: null
[2021-04-14T20:34:02.054Z]   PYTHON_EXE: null
[2021-04-14T20:34:02.054Z]   PYTHON_TEST_FILES: null
[2021-04-14T20:34:02.054Z]   PROCESSES: null
[2021-04-14T20:34:02.054Z]   STRESS_TESTS: null
[2021-04-14T20:34:02.054Z]   STRESS_TEST_OPTIONS: null
[2021-04-14T20:34:02.054Z]   SYSTEM_TESTS: null
[2021-04-14T20:34:02.054Z]   TESTIFY_TOOL_REPO: null
[2021-04-14T20:34:02.054Z]   TEST_ENVIRONMENT: null
[2021-04-14T20:34:02.054Z]   TEST_TAGS: null,oracle
[2021-04-14T20:34:02.054Z]   TESTING_ENVIRONMENT: null
[2021-04-14T20:34:02.054Z]   TIMEOUT: null
[2021-04-14T20:34:02.054Z]   USERPROFILE: C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c
[2021-04-14T20:34:02.054Z]   VENV_PARAMS: null
[2021-04-14T20:34:02.054Z]   XPACK_SUFFIX: null
[2021-04-14T20:34:02.054Z]   ### END ENV DUMP ###
[2021-04-14T20:34:02.054Z]   
[2021-04-14T20:34:02.064Z] Running in C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\heartbeat
[2021-04-14T20:34:02.402Z] 
[2021-04-14T20:34:02.402Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\heartbeat>mage build unitTest 
[2021-04-14T20:34:28.977Z] >> build: Building heartbeat
[2021-04-14T20:35:01.112Z] # github.com/elastic/beats/v7/libbeat/metric/system/process
[2021-04-14T20:35:01.112Z] ..\libbeat\metric\system\process\process.go:571:13: procStats.logger undefined (type *Stats has no field or method logger)
[2021-04-14T20:35:27.720Z] Error: running "go build -o heartbeat.exe -buildmode pie -ldflags -s -X github.com/elastic/beats/v7/libbeat/version.buildTime=2021-04-14T20:34:25Z -X github.com/elastic/beats/v7/libbeat/version.commit=de68b24413882f2d6e92cfcebee1293343a182cc" failed with exit code 2
[2021-04-14T20:35:28.146Z] 
[2021-04-14T20:35:28.146Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats>FOR / %d IN ("ve") DO @IF EXIST "%d" rmdir /s /q "%d" 
[2021-04-14T20:35:28.739Z] 
[2021-04-14T20:35:28.740Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats>python .ci/scripts/pre_archive_test.py 
[2021-04-14T20:35:29.328Z] Running in C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build
[2021-04-14T20:35:29.355Z] Recording test results
[2021-04-14T20:35:29.708Z] None of the test reports contained any result
[2021-04-14T20:35:29.721Z] [Checks API] No suitable checks publisher found.
[2021-04-14T20:35:30.054Z] 
[2021-04-14T20:35:30.054Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build>tar --version  1>NUL 
[2021-04-14T20:35:30.414Z] 
[2021-04-14T20:35:30.414Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build>tar --exclude=test-build-artifacts-heartbeat-windows-10-windows-10-tgz -czf test-build-artifacts-heartbeat-windows-10-windows-10-tgz . 
[2021-04-14T20:35:30.933Z] 
[2021-04-14T20:35:30.933Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build>gsutil --version  1>NUL 
[2021-04-14T20:35:30.933Z] 'gsutil' is not recognized as an internal or external command,
[2021-04-14T20:35:30.933Z] operable program or batch file.
[2021-04-14T20:35:30.991Z] Running in C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build@tmp
[2021-04-14T20:35:31.357Z] 
[2021-04-14T20:35:31.357Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build@tmp>wget --version  1>NUL 
[2021-04-14T20:35:32.004Z] 
[2021-04-14T20:35:32.004Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build@tmp>wget -q -O gsutil.zip https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-319.0.0-windows-x86_64-bundled-python.zip 
[2021-04-14T20:35:37.572Z] Extracting from C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build@tmp\gsutil.zip
[2021-04-14T20:36:00.439Z] Extracted: 15728 files
[2021-04-14T20:36:00.458Z] Masking supported pattern matches of %FILE_CREDENTIAL%
[2021-04-14T20:36:01.499Z] 
[2021-04-14T20:36:01.499Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats\build>gcloud auth activate-service-account --key-file **** 
[2021-04-14T20:36:19.721Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-14T20:36:35.190Z] [INFO] system-tests=''. If no empty then let's create a tarball
[2021-04-14T20:36:35.603Z] 
[2021-04-14T20:36:35.603Z] C:\Users\jenkins\workspace\PR-25095-2-ee9d424e-47e9-43bc-b821-17f9454fd70c\src\github.com\elastic\beats>go clean -modcache 
[2021-04-14T20:38:28.305Z] Failed in branch heartbeat-windows-10-windows-10
[2021-04-14T23:01:47.338Z] Cancelling nested steps due to timeout
[2021-04-14T23:01:47.348Z] Could not connect to beats-ci-immutable-ubuntu-1804-1618431696590264325 to send interrupt signal to process
[2021-04-14T23:01:47.358Z] Sleeping for 5 sec
[2021-04-14T23:01:53.501Z] Failed in branch auditbeat-Lint
[2021-04-14T23:01:53.634Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-14T23:01:53.705Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25095/src/github.com/elastic/beats
[2021-04-14T23:01:54.971Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-25095
[2021-04-14T23:01:55.021Z] [INFO] getVaultSecret: Getting secrets
[2021-04-14T23:01:55.130Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-14T23:01:57.144Z] + chmod 755 generate-build-data.sh
[2021-04-14T23:01:57.144Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25095/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25095/runs/2 ABORTED 11857500
[2021-04-14T23:01:57.144Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25095/runs/2/steps/?limit=10000 -o steps-info.json
[2021-04-14T23:02:08.157Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25095/runs/2/tests/?status=FAILED -o tests-errors.json
[2021-04-14T23:02:10.073Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25095/runs/2/log/ -o pipeline-log.txt

@fearful-symmetry fearful-symmetry requested a review from a team April 14, 2021 22:26
@fearful-symmetry fearful-symmetry merged commit 9d2a24b into elastic:7.x Apr 19, 2021
@zube zube bot removed the [zube]: Done label Jul 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Team:Integrations Label for the Integrations team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants