Skip to content

Cherry-pick #20415 to 7.x: Add host inventory metrics to system module#21204

Merged
kaiyan-sheng merged 1 commit intoelastic:7.xfrom
kaiyan-sheng:backport_20415_7.x
Sep 22, 2020
Merged

Cherry-pick #20415 to 7.x: Add host inventory metrics to system module#21204
kaiyan-sheng merged 1 commit intoelastic:7.xfrom
kaiyan-sheng:backport_20415_7.x

Conversation

@kaiyan-sheng
Copy link
Copy Markdown
Contributor

@kaiyan-sheng kaiyan-sheng commented Sep 21, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #20415 to 7.x branch. Original message:

What does this PR do?

This PR is to add proposed host common fields into system module:

  • host.id
  • host.name
  • host.cpu.pct
  • host.network.in.bytes
  • host.network.in.packets
  • host.network.out.bytes
  • host.network.out.packets
  • host.disk.read.bytes
  • host.disk.write.bytes

For network and disk metrics, instead of reporting one event per interface, host.network.* and host.disk.* aggregates values from all interfaces.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Change system module config system.yml with:

- module: system
  period: 10s
  metricsets:
    - cpu
    - load
    - network
  cpu.metrics: [normalized_percentages]

Then start Metricbeat with ./metricbeat -e and you should see host common fields listed above in events.

Event Example

CPU metric:

{
    "event": {
      "duration": 88523,
      "dataset": "system.cpu",
      "module": "system"
    },
    "metricset": {
      "name": "cpu",
      "period": 10000
    },
    "service": {
      "type": "system"
    },
    "system": {
      "cpu": {
        "user": {
          "pct": 0.7653,
          "norm": {
            "pct": 0.0638
          }
        },
        "steal": {
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "cores": 12,
        "system": {
          "pct": 0.2961,
          "norm": {
            "pct": 0.0247
          }
        },
        "iowait": {
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "nice": {
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "total": {
          "norm": {
            "pct": 0.0884
          },
          "pct": 1.0614
        },
        "idle": {
          "pct": 10.9386,
          "norm": {
            "pct": 0.9116
          }
        },
        "irq": {
          "norm": {
            "pct": 0
          },
          "pct": 0
        },
        "softirq": {
          "pct": 0,
          "norm": {
            "pct": 0
          }
        }
      }
    },
    "host": {
      "cpu": {
        "pct": 0.0884
      },
      "name": "KaiyanMacBookPro",
      "hostname": "KaiyanMacBookPro",
      "architecture": "x86_64"
    }
  }
}

Network metrics:

{
  "_source": {
    "metricset": {
      "name": "network",
      "period": 60000
    },
    "service": {
      "type": "system"
    },
    "host.network.out.bytes": 26123693254,
    "host.network.out.packets": 134348248,
    "event": {
      "dataset": "system.network",
      "module": "system",
      "duration": 6234971
    },
    "host.network.in.bytes": 74833783286,
    "host.network.in.packets": 139823676,
    "host": {
      "name": "KaiyanMacBookPro",
      "hostname": "KaiyanMacBookPro",
      "architecture": "x86_64",
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D"
    }
}

Disk metrics:

{
  "_source": {
    "host.disk.read.bytes": 535049839616,
    "host.disk.write.bytes": 782890594304,
    "event": {
      "dataset": "system.diskio",
      "module": "system",
      "duration": 606341
    },
    "metricset": {
      "name": "diskio",
      "period": 10000
    },
    "service": {
      "type": "system"
    },
    "host": {
      "name": "KaiyanMacBookPro",
      "architecture": "x86_64",
      "hostname": "KaiyanMacBookPro"
    }
  }
}

Related issues

Add host inventory metrics to system module (#20415)
e0c56ba 
* Add host inventory metrics to system module
* Update data.json files

(cherry picked from commit 4a27562)
@kaiyan-sheng kaiyan-sheng added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Sep 21, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 21, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 21, 2020

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 21, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Sep 21, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Sep 21, 2020

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21204 opened]

  • Start Time: 2020-09-21T21:40:29.334+0000

  • Duration: 77 min 9 sec

Test stats 🧪

Test Results
Failed 0
Passed 3211
Skipped 704
Total 3915

@kaiyan-sheng kaiyan-sheng merged commit 6d8ef8d into elastic:7.x Sep 22, 2020
@kaiyan-sheng kaiyan-sheng deleted the backport_20415_7.x branch September 22, 2020 13:17
@zube zube bot removed the [zube]: Done label Dec 22, 2020
@dedemorton dedemorton mentioned this pull request Jan 28, 2021
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ChrsMark ChrsMark ChrsMark approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

backport Team:Platforms Label for the Integrations - Platforms team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kaiyan-sheng @elasticmachine @ChrsMark