[Metricbeat] Add Linux pressure metricset by b-deam · Pull Request #27355 · elastic/beats

b-deam · 2021-08-13T05:29:07Z

What does this PR do?

This PR aims to introduce an additional metricset that collects Pressure Stall Information (PSI) for CPU, Memory, and IO. These are available from kernel >=4.20.

These metrics are incredibly useful for debugging system performance, and more can be read about these metrics below:

Why is it important?

These are valuable OS metrics that are useful for both debugging and monitoring system performance.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Tests need work.

How to test this PR locally

Pull down, run on a Linux host (ubuntu2004 works from Vagrantfile) that:
- Is on kernel >=4.20
- /proc/pressure is available on

Related issues

Closes [Metricbeat][System module] psi: pressure stall information #27223

…tricset

elasticmachine · 2021-08-13T05:44:04Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-19T03:21:52.887+0000
Duration: 81 min 24 sec
Commit: c4cbb2a

Test stats 🧪

Test	Results
Failed	0
Passed	9086
Skipped	2418
Total	11504

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9086
Skipped	2418
Total	11504

fearful-symmetry

This is looking good! Just a few small things, mostly beats cruft, we need to change. If you want, I can take the PR over from here; I don't want to make you feel like you're obligated to spend your time coding to the various metricbeat idioms.

fearful-symmetry · 2021-08-13T16:42:10Z

metricbeat/module/linux/pressure/_meta/fields.yml

+      type: group
+      description: Linux pressure stall information
+      fields:
+        - name: cpu.some.10


In system/process cgroup data, we're reporting these as 10.pct and so on. We'll also need to add format: percent

Cool. I tried to follow similar conventions for total, in that it becomes total.time.us - does this make sense here?

metricbeat/module/linux/pressure/pressure.go

metricbeat/module/linux/pressure/pressure_test.go

b-deam · 2021-08-16T08:48:00Z

If you want, I can take the PR over from here; I don't want to make you feel like you're obligated to spend your time coding to the various metricbeat idioms.

Not a problem - I think I've addressed the comments so far but feel free to make any small changes if I've otherwise messed things up there.

fearful-symmetry · 2021-08-16T16:06:57Z

Real quick, you'll need to run mage fmt update in the main metricbeat directory.

b-deam · 2021-08-16T23:05:13Z

Real quick, you'll need to run mage fmt update in the main metricbeat directory.

Hmm.. I tried this but I don't see any changes. Any ideas?

bradleydeam in ~/go/src/github.com/b-deam/beats/metricbeat on branch linux-pressure-metricset $ make update  
mage update
Generated fields.yml for metricbeat to /Users/bradleydeam/go/src/github.com/b-deam/beats/metricbeat/fields.yml
>> Building metricbeat.yml for linux/amd64
>> Building metricbeat.reference.yml for linux/amd64
>> Building metricbeat.docker.yml for linux/amd64
Generated fields.yml for metricbeat to /Users/bradleydeam/go/src/github.com/b-deam/beats/metricbeat/build/fields/fields.all.yml
exec: go list -m
exec: go list -m              

bradleydeam in ~/go/src/github.com/b-deam/beats/metricbeat on branch linux-pressure-metricset $ mage fmt update
>> fmt - go-licenser: Adding missing headers
>> fmt - goimports: Formatting Go code
>> fmt - autopep8: Formatting Python code
Generated fields.yml for metricbeat to /Users/bradleydeam/go/src/github.com/b-deam/beats/metricbeat/fields.yml
>> Building metricbeat.yml for linux/amd64
>> Building metricbeat.reference.yml for linux/amd64
>> Building metricbeat.docker.yml for linux/amd64
Generated fields.yml for metricbeat to /Users/bradleydeam/go/src/github.com/b-deam/beats/metricbeat/build/fields/fields.all.yml
exec: go list -m
exec: go list -m

bradleydeam in ~/go/src/github.com/b-deam/beats/metricbeat on branch linux-pressure-metricset $ git status
On branch linux-pressure-metricset
Your branch is up to date with 'origin/linux-pressure-metricset'.

nothing to commit, working tree clean

fearful-symmetry · 2021-08-17T16:27:19Z

@b-deam I think you'll need to re-merge upstream changes? A few things may have changed in the doc build.

…tricset

b-deam · 2021-08-18T01:48:12Z

@b-deam I think you'll need to re-merge upstream changes? A few things may have changed in the doc build.

🤦 - ofc. Done. 🤞 tests are good!

elasticmachine · 2021-08-18T05:25:20Z

Pinging @elastic/integrations (Team:Integrations)

metricbeat/module/linux/pressure/pressure.go

metricbeat/module/linux/pressure/pressure_test.go

fearful-symmetry

LGTM.

…tricset

b-deam · 2021-08-19T03:22:26Z

LGTM.

I re-ran a test to make sure it all looked good and noticed that I'd messed up the fields/mapping so that the metrics were actually recorded as:

linux.pressure.pressure.*

This should be fixed now..

b-deam · 2021-08-19T05:04:10Z

Thanks for all the help @fearful-symmetry!

* Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping

* Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping (cherry picked from commit e49f257) # Conflicts: # metricbeat/docs/modules_list.asciidoc # metricbeat/module/linux/fields.go # metricbeat/module/linux/pressure/pressure.go

* Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping (cherry picked from commit e49f257) # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/docs/modules/linux.asciidoc # metricbeat/docs/modules/linux/pressure.asciidoc # metricbeat/docs/modules_list.asciidoc # metricbeat/include/list_common.go # metricbeat/metricbeat.reference.yml # metricbeat/module/linux/_meta/config.yml # metricbeat/module/linux/fields.go # metricbeat/module/linux/pressure/pressure.go # metricbeat/module/linux/pressure/pressure_test.go # metricbeat/modules.d/linux.yml.disabled # x-pack/metricbeat/metricbeat.reference.yml

) * [Metricbeat] Add Linux pressure metricset (#27355) * Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping (cherry picked from commit e49f257) # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/docs/modules/linux.asciidoc # metricbeat/docs/modules/linux/pressure.asciidoc # metricbeat/docs/modules_list.asciidoc # metricbeat/include/list_common.go # metricbeat/metricbeat.reference.yml # metricbeat/module/linux/_meta/config.yml # metricbeat/module/linux/fields.go # metricbeat/module/linux/pressure/pressure.go # metricbeat/module/linux/pressure/pressure_test.go # metricbeat/modules.d/linux.yml.disabled # x-pack/metricbeat/metricbeat.reference.yml * fix merge Co-authored-by: Brad Deam <54515790+b-deam@users.noreply.github.com> Co-authored-by: Alex Kristiansen <alex.kristiansen@elastic.co>

…9036) * [Metricbeat] Add Linux pressure metricset (#27355) * Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping (cherry picked from commit e49f257) # Conflicts: # metricbeat/docs/modules_list.asciidoc # metricbeat/module/linux/fields.go # metricbeat/module/linux/pressure/pressure.go * Fix conflicts * Formatting Co-authored-by: Brad Deam <54515790+b-deam@users.noreply.github.com> Co-authored-by: Brad Deam <bradley.deam@elastic.co>

b-deam added 5 commits August 12, 2021 17:47

Adds new linux pressure metricset

425ef9b

Tidy up code

c6a1ce0

Merge remote-tracking branch 'upstream/master' into linux-pressure-me…

baa4dc9

…tricset

Be consistent in default config

b63496b

Run integration tests

f3f531a

b-deam added enhancement Metricbeat Metricbeat Team:Integrations Label for the Integrations team :integrations labels Aug 13, 2021

b-deam self-assigned this Aug 13, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 13, 2021

b-deam changed the title ~~Linux pressure metricset~~ [Metricbeat] Add Linux pressure metricset Aug 13, 2021

b-deam requested a review from fearful-symmetry August 13, 2021 05:29

b-deam added 2 commits August 13, 2021 16:19

Make update

4da2423

make update v3

4d572d6

fearful-symmetry reviewed Aug 13, 2021

View reviewed changes

fearful-symmetry self-assigned this Aug 13, 2021

b-deam added 3 commits August 16, 2021 17:59

Address comments

a470b1e

Format tests

2c41126

Remove unused imports from tests

d46e1ea

b-deam added 2 commits August 18, 2021 11:14

Merge remote-tracking branch 'upstream/master' into linux-pressure-me…

e4c39dd

…tricset

Mage format

ac5f0ee

Update changelog

b496aeb

b-deam marked this pull request as ready for review August 18, 2021 05:25

fearful-symmetry reviewed Aug 18, 2021

View reviewed changes

metricbeat/module/linux/pressure/pressure.go Show resolved Hide resolved

fearful-symmetry reviewed Aug 18, 2021

View reviewed changes

metricbeat/module/linux/pressure/pressure_test.go Outdated Show resolved Hide resolved

fearful-symmetry approved these changes Aug 18, 2021

View reviewed changes

b-deam added 3 commits August 19, 2021 09:18

Merge remote-tracking branch 'upstream/master' into linux-pressure-me…

4afa1f6

…tricset

simplify loop

4451182

Fix fields/mapping

c4cbb2a

b-deam merged commit e49f257 into elastic:master Aug 19, 2021

andresrc added backport-v7.16.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify labels Nov 18, 2021

mergify bot mentioned this pull request Nov 18, 2021

[7.16](backport #27355) [Metricbeat] Add Linux pressure metricset #29036

Merged

mergify bot mentioned this pull request Nov 18, 2021

[8.0](backport #27355) [Metricbeat] Add Linux pressure metricset #29037

Merged

Conversation

b-deam commented Aug 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Aug 13, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

fearful-symmetry left a comment

Choose a reason for hiding this comment

Uh oh!

fearful-symmetry Aug 13, 2021

Choose a reason for hiding this comment

Uh oh!

b-deam Aug 16, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

b-deam commented Aug 16, 2021

Uh oh!

fearful-symmetry commented Aug 16, 2021

Uh oh!

b-deam commented Aug 16, 2021

Uh oh!

fearful-symmetry commented Aug 17, 2021

Uh oh!

b-deam commented Aug 18, 2021

Uh oh!

elasticmachine commented Aug 18, 2021

Uh oh!

Uh oh!

Uh oh!

fearful-symmetry left a comment

Choose a reason for hiding this comment

Uh oh!

b-deam commented Aug 19, 2021

Uh oh!

b-deam commented Aug 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

b-deam commented Aug 13, 2021 •

edited

Loading

elasticmachine commented Aug 13, 2021 •

edited by jenkins-beats-ci bot

Loading