Skip to content

metricbeat: collect total thread count per process#36578

Merged
AndersonQ merged 2 commits intoelastic:mainfrom
AndersonQ:34461-collect-total-thread-count
Oct 4, 2023
Merged

metricbeat: collect total thread count per process#36578
AndersonQ merged 2 commits intoelastic:mainfrom
AndersonQ:34461-collect-total-thread-count

Conversation

@AndersonQ
Copy link
Copy Markdown
Member

@AndersonQ AndersonQ commented Sep 13, 2023 •
edited
Loading

Collect the total by thread count per process upgrading elastic-agent-system-metrics to v0.7.0

Proposed commit message

metricbeat: collect total thread count per process

Upgrades elastic-agent-system-metrics to v0.5.0 in order to collect the total thread count per process.

Checklist

  • My code follows the style guidelines of this project
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Logs

❯ go test -v -run TestFetch  ./metricbeat/module/system/process
=== RUN   TestFetch

// debug logs omitted
 
    process_test.go:51: fetched 5 events, showing events[0]:
    process_test.go:52: system/process event: {
          "process": {
            "args": [
              "/tmp/go-build658655655/b001/process.test",
              "-test.testlogfile=/tmp/go-build658655655/b001/testlog.txt",
              "-test.paniconexit0",
              "-test.timeout=10m0s",
              "-test.v=true",
              "-test.run=TestFetch"
            ],
            "command_line": "/tmp/go-build658655655/b001/process.test -test.testlogfile=/tmp/go-build658655655/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true -test.run=TestFetch",
            "cpu": {
              "pct": 0.0171,
              "start_time": "2023-09-13T11:52:45.000Z"
            },
            "executable": "/tmp/go-build658655655/b001/process.test",
            "memory": {
              "pct": 0.0004
            },
            "name": "process.test",
            "parent": {
              "pid": 196866
            },
            "pgid": 196866,
            "pid": 197122,
            "state": "sleeping",
            "working_directory": "/home/ainsoph/devel/github.com/elastic/beats/metricbeat/module/system/process"
          },
          "service": {
            "type": "system"
          },
          "system": {
            "process": {
              "cgroup": {
                "cgroups_version": 2,
                "cpu": {
                  "id": "vte-spawn-fa31675c-a919-4c0a-9447-9fdc56de8544.scope",
                  "path": "/user.slice/user-1000.slice/user@1000.service/app.slice/vte-spawn-fa31675c-a919-4c0a-9447-9fdc56de8544.scope",
                  "pressure": {
                    "full": {
                      "10": {
                        "pct": 0
                      },
                      "300": {
                        "pct": 0
                      },
                      "60": {
                        "pct": 0
                      },
                      "total": 4383214
                    },
                    "some": {
                      "10": {
                        "pct": 0
                      },
                      "300": {
                        "pct": 0
                      },
                      "60": {
                        "pct": 0
                      },
                      "total": 4588411
                    }
                  },
                  "stats": {
                    "system": {
                      "norm": {
                        "pct": 0
                      },
                      "ns": 30786963,
                      "pct": 0.0002
                    },
                    "usage": {
                      "norm": {
                        "pct": 0
                      },
                      "ns": 54487651,
                      "pct": 0.0003
                    },
                    "user": {
                      "norm": {
                        "pct": 0
                      },
                      "ns": 23700688,
                      "pct": 0.0001
                    }
                  }
                },
                "id": "vte-spawn-fa31675c-a919-4c0a-9447-9fdc56de8544.scope",
                "memory": {
                  "id": "vte-spawn-fa31675c-a919-4c0a-9447-9fdc56de8544.scope",
                  "mem": {
                    "events": {
                      "high": 0,
                      "low": 0,
                      "max": 0,
                      "oom": 0,
                      "oom_kill": 0
                    },
                    "low": {
                      "bytes": 0
                    },
                    "usage": {
                      "bytes": 551112704
                    }
                  },
                  "memsw": {
                    "events": {
                      "fail": 0,
                      "high": 0,
                      "max": 0
                    },
                    "low": {
                      "bytes": 0
                    },
                    "usage": {
                      "bytes": 0
                    }
                  },
                  "path": "/user.slice/user-1000.slice/user@1000.service/app.slice/vte-spawn-fa31675c-a919-4c0a-9447-9fdc56de8544.scope",
                  "stats": {
                    "active_anon": {
                      "bytes": 74604544
                    },
                    "active_file": {
                      "bytes": 161099776
                    },
                    "anon": {
                      "bytes": 64045056
                    },
                    "anon_thp": {
                      "bytes": 14680064
                    },
                    "file": {
                      "bytes": 400678912
                    },
                    "file_dirty": {
                      "bytes": 34336768
                    },
                    "file_mapped": {
                      "bytes": 38699008
                    },
                    "file_thp": {
                      "bytes": 0
                    },
                    "file_writeback": {
                      "bytes": 0
                    },
                    "htp_collapse_alloc": 0,
                    "inactive_anon": {
                      "bytes": 24576
                    },
                    "inactive_file": {
                      "bytes": 239509504
                    },
                    "kernel_stack": {
                      "bytes": 1376256
                    },
                    "major_page_faults": 1666,
                    "page_activate": 0,
                    "page_deactivate": 0,
                    "page_faults": 987725,
                    "page_lazy_free": 0,
                    "page_lazy_freed": 0,
                    "page_refill": 0,
                    "page_scan": 0,
                    "page_steal": 0,
                    "page_tables": {
                      "bytes": 835584
                    },
                    "per_cpu": {
                      "bytes": 1728
                    },
                    "shmem": {
                      "bytes": 0
                    },
                    "shmem_thp": {
                      "bytes": 0
                    },
                    "slab": {
                      "bytes": 71222912
                    },
                    "slab_reclaimable": {
                      "bytes": 69878216
                    },
                    "slab_unreclaimable": {
                      "bytes": 1344696
                    },
                    "sock": {
                      "bytes": 0
                    },
                    "swap_cached": {
                      "bytes": 0
                    },
                    "thp_fault_alloc": 876,
                    "unevictable": {
                      "bytes": 0
                    },
                    "workingset_activate_anon": 0,
                    "workingset_activate_file": 0,
                    "workingset_node_reclaim": 0,
                    "workingset_refault_anon": 0,
                    "workingset_refault_file": 0,
                    "workingset_restore_anon": 0,
                    "workingset_restore_file": 0
                  }
                },
                "path": "/user.slice/user-1000.slice/user@1000.service/app.slice/vte-spawn-fa31675c-a919-4c0a-9447-9fdc56de8544.scope"
              },
              "cmdline": "/tmp/go-build658655655/b001/process.test -test.testlogfile=/tmp/go-build658655655/b001/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true -test.run=TestFetch",
              "cpu": {
                "start_time": "2023-09-13T11:52:45.000Z",
                "system": {
                  "ticks": 930
                },
                "total": {
                  "norm": {
                    "pct": 0.0171
                  },
                  "pct": 0.2743,
                  "ticks": 1420,
                  "value": 1420
                },
                "user": {
                  "ticks": 490
                }
              },
              "fd": {
                "limit": {
                  "hard": 1048576,
                  "soft": 1048576
                },
                "open": 9
              },
              "memory": {
                "rss": {
                  "bytes": 29417472,
                  "pct": 0.0004
                },
                "share": 16908288,
                "size": 2642882560
              },
              "num_threads": 19,
              "state": "sleeping"
            }
          },
          "user": {
            "name": "ainsoph"
          }
        }
--- PASS: TestFetch (3.26s)
PASS
ok  	github.com/elastic/beats/v7/metricbeat/module/system/process	3.569s

@AndersonQ AndersonQ added enhancement Team:Elastic-Agent Label for the Agent team backport-skip Skip notification from the automated backport with mergify labels Sep 13, 2023
@AndersonQ AndersonQ self-assigned this Sep 13, 2023
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 13, 2023
@AndersonQ AndersonQ marked this pull request as ready for review September 13, 2023 11:57
@AndersonQ AndersonQ requested review from a team as code owners September 13, 2023 11:57
AndersonQ
AndersonQ commented Sep 13, 2023
View reviewed changes
go.mod Outdated
// Exclude this version because the version has an invalid checksum.
exclude github.com/docker/distribution v2.8.0+incompatible

replace github.com/elastic/elastic-agent-system-metrics => github.com/AndersonQ/elastic-agent-system-metrics v0.0.0-20230913114254-ef72b6fa59d9
Copy link
Copy Markdown
Member Author

@AndersonQ AndersonQ Sep 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As soon as elastic/elastic-agent-system-metrics#96 gets merged and elastic-agent-system-metrics v0.5.0 released, I'll remove the replace and properly upgrade elastic-agent-system-metrics to v0.5.0.

@AndersonQ AndersonQ force-pushed the 34461-collect-total-thread-count branch from 95dfb9c to 07bfb27 Compare September 13, 2023 12:09
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 13, 2023 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-02T21:40:52.406+0000

  • Duration: 117 min 29 sec

Test stats 🧪

Test Results
Failed 0
Passed 27832
Skipped 1880
Total 29712

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@AndersonQ AndersonQ force-pushed the 34461-collect-total-thread-count branch from 07bfb27 to 14404e7 Compare September 14, 2023 05:30
belimawr
belimawr previously requested changes Sep 14, 2023
View reviewed changes
libbeat/tests/system/beat/beat.py Outdated
Comment on lines +680 to +682
# print("===================================================================================================")
# print("doc_list:", doc_list)
# print("===================================================================================================")
Copy link
Copy Markdown
Contributor

@belimawr belimawr Sep 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug comments?

Suggested change
# print("===================================================================================================")
# print("doc_list:", doc_list)
# print("===================================================================================================")

libbeat/tests/system/beat/beat.py Outdated
Comment on lines +723 to +726

print( "=================================================================================================")
print("path:", path)
print( "=================================================================================================")
Copy link
Copy Markdown
Contributor

@belimawr belimawr Sep 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug comments?

Suggested change
print( "=================================================================================================")
print("path:", path)
print( "=================================================================================================")

@AndersonQ AndersonQ force-pushed the 34461-collect-total-thread-count branch 2 times, most recently from 44aea18 to 18e0a67 Compare September 26, 2023 06:13
@AndersonQ AndersonQ force-pushed the 34461-collect-total-thread-count branch 4 times, most recently from 98b2b4d to fa6ad19 Compare September 29, 2023 15:27
@AndersonQ
Copy link
Copy Markdown
Member Author

AndersonQ commented Sep 29, 2023

/test

@AndersonQ
metricbeat: collect number of threads per process
8ec8800 
Upgrades elastic-agent-system-metrics to v0.7.0 in order to collect the number of threads per process.
@AndersonQ AndersonQ force-pushed the 34461-collect-total-thread-count branch from fa6ad19 to 8ec8800 Compare October 2, 2023 16:13
@AndersonQ AndersonQ dismissed belimawr’s stale review October 2, 2023 21:39

Alex reviewed it instead

@AndersonQ AndersonQ enabled auto-merge (squash) October 2, 2023 21:40
jlind23
jlind23 approved these changes Oct 4, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@jlind23 jlind23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already approved by @fearful-symmetry

@AndersonQ AndersonQ merged commit 08720a7 into elastic:main Oct 4, 2023
@tetianakravchenko tetianakravchenko mentioned this pull request Dec 21, 2023
Merged
4 tasks
Scholar-Li pushed a commit to Scholar-Li/beats that referenced this pull request Feb 5, 2024
@AndersonQ @Scholar-Li
metricbeat: collect total thread count per process (elastic#36578)
bd0ea1b 
* metricbeat: collect number of threads per process

Upgrades elastic-agent-system-metrics to v0.7.0 in order to collect the number of threads per process.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fearful-symmetry fearful-symmetry fearful-symmetry approved these changes

@jlind23 jlind23 jlind23 approved these changes

@belimawr belimawr belimawr left review comments

@ycombinator ycombinator Awaiting requested review from ycombinator ycombinator is a code owner automatically assigned from elastic/elastic-agent-data-plane

Assignees

@AndersonQ AndersonQ

Labels

backport-skip Skip notification from the automated backport with mergify enhancement Team:Elastic-Agent Label for the Agent team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Metricbeat system module - collect total thread count

5 participants

@AndersonQ @elasticmachine @belimawr @fearful-symmetry @jlind23