Skip to content

[system][process] Enable tsdb for process datastream#7672

Merged
tetianakravchenko merged 5 commits intoelastic:mainfrom
tetianakravchenko:system-process-enable-tsdb
Oct 17, 2023
Merged

[system][process] Enable tsdb for process datastream#7672
tetianakravchenko merged 5 commits intoelastic:mainfrom
tetianakravchenko:system-process-enable-tsdb

Conversation

@tetianakravchenko
Copy link
Copy Markdown
Contributor

@tetianakravchenko tetianakravchenko commented Sep 6, 2023 •
edited
Loading

What does this PR do?

Enable tsdb for process datastream. waiting for this PR - #7660 to be merged first

Tested with TSDB-migration-test-kit:

Details 
Testing data stream metrics-system.process-default.
Index being used for the documents is .ds-metrics-system.process-default-2023.09.05-000001.
Index being used for the settings and mappings is .ds-metrics-system.process-default-2023.09.05-000001.

The time series fields for the TSDB index are: 
        - dimension (9 fields):
                - agent.id
                - cloud.account.id
                - cloud.availability_zone
                - cloud.instance.id
                - cloud.provider
                - cloud.region
                - container.id
                - host.name
                - process.pid
        - counter (40 fields):
                - system.process.cgroup.blkio.total.bytes
                - system.process.cgroup.blkio.total.ios
                - system.process.cgroup.cpu.pressure.full.total
                - system.process.cgroup.cpu.pressure.some.total
                - system.process.cgroup.cpu.stats.periods
                - system.process.cgroup.cpu.stats.system.ns
                - system.process.cgroup.cpu.stats.throttled.ns
                - system.process.cgroup.cpu.stats.throttled.periods
                - system.process.cgroup.cpu.stats.throttled.us
                - system.process.cgroup.cpu.stats.usage.ns
                - system.process.cgroup.cpu.stats.user.ns
                - system.process.cgroup.cpuacct.stats.system.ns
                - system.process.cgroup.cpuacct.stats.user.ns
                - system.process.cgroup.cpuacct.total.ns
                - system.process.cgroup.io.pressure.full.total
                - system.process.cgroup.io.pressure.some.total
                - system.process.cgroup.memory.kmem.failures
                - system.process.cgroup.memory.kmem_tcp.failures
                - system.process.cgroup.memory.mem.events.fail
                - system.process.cgroup.memory.mem.events.high
                - system.process.cgroup.memory.mem.events.low
                - system.process.cgroup.memory.mem.events.max
                - system.process.cgroup.memory.mem.events.oom
                - system.process.cgroup.memory.mem.events.oom_kill
                - system.process.cgroup.memory.mem.failures
                - system.process.cgroup.memory.memsw.events.fail
                - system.process.cgroup.memory.memsw.events.high
                - system.process.cgroup.memory.memsw.events.low
                - system.process.cgroup.memory.memsw.events.max
                - system.process.cgroup.memory.memsw.events.oom
                - system.process.cgroup.memory.memsw.events.oom_kill
                - system.process.cgroup.memory.memsw.failures
                - system.process.cgroup.memory.stats.major_page_faults
                - system.process.cgroup.memory.stats.page_faults
                - system.process.cgroup.memory.stats.pages_in
                - system.process.cgroup.memory.stats.pages_out
                - system.process.cpu.system.ticks
                - system.process.cpu.total.ticks
                - system.process.cpu.total.value
                - system.process.cpu.user.ticks
        - gauge (68 fields):
                - system.process.cgroup.cpu.cfs.period.us
                - system.process.cgroup.cpu.cfs.quota.us
                - system.process.cgroup.cpu.cfs.shares
                - system.process.cgroup.cpu.pressure.full.10.pct
                - system.process.cgroup.cpu.pressure.full.300.pct
                - system.process.cgroup.cpu.pressure.full.60.pct
                - system.process.cgroup.cpu.pressure.some.10.pct
                - system.process.cgroup.cpu.pressure.some.300.pct
                - system.process.cgroup.cpu.pressure.some.60.pct
                - system.process.cgroup.cpu.rt.period.us
                - system.process.cgroup.cpu.rt.runtime.us
                - system.process.cgroup.cpu.stats.system.norm.pct
                - system.process.cgroup.cpu.stats.system.pct
                - system.process.cgroup.cpu.stats.usage.norm.pct
                - system.process.cgroup.cpu.stats.usage.pct
                - system.process.cgroup.cpu.stats.user.norm.pct
                - system.process.cgroup.cpu.stats.user.pct
                - system.process.cgroup.cpuacct.stats.system.norm.pct
                - system.process.cgroup.cpuacct.stats.system.pct
                - system.process.cgroup.cpuacct.stats.user.norm.pct
                - system.process.cgroup.cpuacct.stats.user.pct
                - system.process.cgroup.cpuacct.total.norm.pct
                - system.process.cgroup.cpuacct.total.pct
                - system.process.cgroup.io.pressure.full.10.pct
                - system.process.cgroup.io.pressure.full.300.pct
                - system.process.cgroup.io.pressure.full.60.pct
                - system.process.cgroup.io.pressure.some.10.pct
                - system.process.cgroup.io.pressure.some.300.pct
                - system.process.cgroup.io.pressure.some.60.pct
                - system.process.cgroup.memory.kmem.limit.bytes
                - system.process.cgroup.memory.kmem.usage.bytes
                - system.process.cgroup.memory.kmem.usage.max.bytes
                - system.process.cgroup.memory.kmem_tcp.limit.bytes
                - system.process.cgroup.memory.kmem_tcp.usage.bytes
                - system.process.cgroup.memory.kmem_tcp.usage.max.bytes
                - system.process.cgroup.memory.mem.high.bytes
                - system.process.cgroup.memory.mem.limit.bytes
                - system.process.cgroup.memory.mem.low.bytes
                - system.process.cgroup.memory.mem.max.bytes
                - system.process.cgroup.memory.mem.usage.bytes
                - system.process.cgroup.memory.mem.usage.max.bytes
                - system.process.cgroup.memory.memsw.high.bytes
                - system.process.cgroup.memory.memsw.limit.bytes
                - system.process.cgroup.memory.memsw.low.bytes
                - system.process.cgroup.memory.memsw.max.bytes
                - system.process.cgroup.memory.memsw.usage.bytes
                - system.process.cgroup.memory.memsw.usage.max.bytes
                - system.process.cgroup.memory.stats.active_anon.bytes
                - system.process.cgroup.memory.stats.active_file.bytes
                - system.process.cgroup.memory.stats.cache.bytes
                - system.process.cgroup.memory.stats.hierarchical_memory_limit.bytes
                - system.process.cgroup.memory.stats.hierarchical_memsw_limit.bytes
                - system.process.cgroup.memory.stats.inactive_anon.bytes
                - system.process.cgroup.memory.stats.inactive_file.bytes
                - system.process.cgroup.memory.stats.mapped_file.bytes
                - system.process.cgroup.memory.stats.rss.bytes
                - system.process.cgroup.memory.stats.rss_huge.bytes
                - system.process.cgroup.memory.stats.swap.bytes
                - system.process.cgroup.memory.stats.unevictable.bytes
                - system.process.cpu.total.norm.pct
                - system.process.cpu.total.pct
                - system.process.fd.limit.hard
                - system.process.fd.limit.soft
                - system.process.fd.open
                - system.process.memory.rss.bytes
                - system.process.memory.rss.pct
                - system.process.memory.share
                - system.process.memory.size
        - routing_path (8 fields):
                - agent.id
                - cloud.account.id
                - cloud.availability_zone
                - cloud.instance.id
                - cloud.provider
                - cloud.region
                - container.id
                - host.name

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-system.process-default-2023.09.05-000001 to tsdb-index-enabled...
All 6540 documents taken from index .ds-metrics-system.process-default-2023.09.05-000001 were successfully placed to index tsdb-index-enabled.

Note: list of routing_path does not include process.pid, even though it is a dimension field.

Process information is available on the host dashboard after enabling tsdb:
Screenshot 2023-09-06 at 11 11 08

And available on the inventory page:
Screenshot 2023-09-06 at 11 09 00

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@tetianakravchenko
enable tsdb for process datastream
a272159 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko tetianakravchenko requested review from a team as code owners September 6, 2023 09:12
@tetianakravchenko tetianakravchenko mentioned this pull request Sep 6, 2023
Closed
11 tasks
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Sep 6, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-17T14:49:57.319+0000

  • Duration: 17 min 23 sec

Test stats 🧪

Test Results
Failed 0
Passed 151
Skipped 0
Total 151

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Sep 6, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (3/3) 💚
Files 100.0% (4/4) 💚 5.556
Classes 100.0% (4/4) 💚 5.556
Methods 63.415% (52/82) 👎 -32.275
Lines 99.863% (2924/2928) 👍 13.78
Conditionals 100.0% (0/0) 💚

@botelastic
Copy link
Copy Markdown

botelastic bot commented Oct 6, 2023

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Oct 6, 2023
@botelastic botelastic bot removed the Stalled label Oct 10, 2023
ritalwar
ritalwar approved these changes Oct 17, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@ritalwar ritalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tetianakravchenko
Merge branch 'main' into system-process-enable-tsdb
62e8444
@tetianakravchenko
update the package version
5d67d73 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko tetianakravchenko merged commit cc7f38a into elastic:main Oct 17, 2023
@tetianakravchenko tetianakravchenko deleted the system-process-enable-tsdb branch October 17, 2023 15:22
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 17, 2023

Package system - 1.44.0 containing this change is available at https://epr.elastic.co/search?package=system

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ritalwar ritalwar ritalwar approved these changes

@lalit-satapathy lalit-satapathy Awaiting requested review from lalit-satapathy

@agithomas agithomas Awaiting requested review from agithomas

Assignees

No one assigned

Labels

Integration:system System

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tetianakravchenko @elasticmachine @ritalwar @andrewkroh