Skip to content

[Fleet] Support dataset with multiple level like system.process.summary#99852

Merged
nchaulet merged 3 commits intoelastic:masterfrom
nchaulet:fix-92636-support-dataset-dots
May 12, 2021
Merged

[Fleet] Support dataset with multiple level like system.process.summary#99852
nchaulet merged 3 commits intoelastic:masterfrom
nchaulet:fix-92636-support-dataset-dots

Conversation

@nchaulet
Copy link
Copy Markdown
Member

@nchaulet nchaulet commented May 11, 2021 •
edited
Loading

Summary

Resolve #92636

Currently when a datastream use a dataset with multiple dots we are not using the correct agent template.
We where looking to find the dastream by splitting the dataset name on . instead of using the datastream path property
That PR fix that.

How to test

You can try that PR elastic/integrations#778 run the package registry and you will get the correct
agent template for system.process.summary

Screen Shot 2021-05-11 at 6 01 20 PM

@nchaulet nchaulet added bug Fixes for quality problems that affect the customer experience v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.14.0 labels May 11, 2021
@nchaulet nchaulet self-assigned this May 11, 2021
@nchaulet nchaulet requested a review from a team as a code owner May 11, 2021 21:44
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented May 11, 2021

Pinging @elastic/fleet (Team:Fleet)

@nchaulet nchaulet requested a review from jen-huang May 11, 2021 22:02
@nchaulet
Copy link
Copy Markdown
Member Author

nchaulet commented May 11, 2021

Should we backport this to 7.13?

jen-huang
jen-huang approved these changes May 11, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@jen-huang jen-huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Yes, this is a good 7.13 candidate.
  2. Would this be fixed if elastic/integrations#778 also exported path: process_summary? (It's still good to have a fallback in Kibana, aka this PR)

@nchaulet
Copy link
Copy Markdown
Member Author

nchaulet commented May 11, 2021 •
edited
Loading

Would this be fixed if elastic/integrations#778 also exported path: process_summary? (It's still good to have a fallback in Kibana, aka this PR)

Actually the package registry populate the path elastic/package-registry#176 so I think we probably should only rely on that.

Just did a quick check against the snapshot registry and the dataset we computed before and the path match for all packages expect apm endpoint and synthetics where there is no agent data stream template.

Details 
✅ /package/apache/0.5.0
app_metrics app
error_logs error
internal_metrics internal
profile_metrics profiling
sampled_traces
traces
❌ /package/apm/0.2.0
✅ /package/auditd/0.1.1
✅ /package/aws/0.5.4
✅ /package/azure/0.2.3
✅ /package/barracuda/0.2.4
✅ /package/bluecoat/0.1.4
✅ /package/cef/0.1.0
✅ /package/checkpoint/0.5.2
✅ /package/cisco/0.8.1
✅ /package/crowdstrike/0.4.1
✅ /package/cyberark/0.1.4
✅ /package/cylance/0.1.4
✅ /package/docker/0.1.1
✅ /package/elastic_agent/0.0.7
collection diagnostic_collection
file events_file
library events_library
network events_network
process events_process
registry events_registry
security events_security
❌ /package/endpoint/0.19.1
✅ /package/f5/0.2.4
✅ /package/fleet_server/0.2.3
✅ /package/fortinet/0.8.1
✅ /package/gcp/0.0.2
✅ /package/google_workspace/0.2.4
✅ /package/haproxy/0.2.8
✅ /package/iis/0.3.2
✅ /package/imperva/0.1.4
✅ /package/infoblox/0.1.4
✅ /package/iptables/0.1.0
✅ /package/juniper/0.5.1
✅ /package/kafka/0.3.7
✅ /package/kubernetes/0.5.1
✅ /package/linux/0.3.10
✅ /package/log/0.4.6
✅ /package/microsoft/0.4.3
✅ /package/mongodb/0.2.9
✅ /package/mysql/0.3.7
✅ /package/nats/0.1.4
✅ /package/netflow/0.3.8
✅ /package/netscout/0.1.4
✅ /package/nginx/0.5.0
✅ /package/nginx_ingress_controller/0.0.3
✅ /package/o365/0.3.2
✅ /package/okta/0.5.0
✅ /package/osquery/0.1.2
✅ /package/osquery_manager/0.2.2
✅ /package/panw/0.7.1
✅ /package/postgresql/0.3.1
✅ /package/prometheus/0.3.4
✅ /package/proofpoint/0.1.4
✅ /package/rabbitmq/0.2.7
✅ /package/radware/0.1.4
✅ /package/redis/0.3.7
✅ /package/santa/0.0.3
✅ /package/security_detection_engine/0.0.2
✅ /package/snapshot/0.0.1
✅ /package/sonicwall/0.1.5
✅ /package/sophos/0.2.1
✅ /package/squid/0.1.4
✅ /package/staging/0.0.1
✅ /package/stan/0.1.3
✅ /package/suricata/0.6.0
✅ /package/symantec/0.1.2
http
icmp
tcp
❌ /package/synthetics/0.0.4
✅ /package/system/0.12.3
✅ /package/tomcat/0.1.4
✅ /package/traefik/0.1.1
✅ /package/windows/0.7.0
✅ /package/winlog/0.2.1
✅ /package/zeek/0.7.0
✅ /package/zookeeper/0.2.6
✅ /package/zoom/0.3.1
✅ /package/zscaler/0.1.4

@nchaulet nchaulet added the auto-backport Deprecated - use backport:version if exact versions are needed label May 11, 2021
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 12, 2021

💚 Build Succeeded

Metrics [docs]

Unknown metric groups

References to deprecated APIs

id before after diff
infra 5 3 -2
monitoring 109 56 -53
total -55

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

@nchaulet
Copy link
Copy Markdown
Member Author

nchaulet commented May 12, 2021

/run-fleet-e2e-tests

@nchaulet nchaulet merged commit ae08154 into elastic:master May 12, 2021
@nchaulet nchaulet deleted the fix-92636-support-dataset-dots branch May 12, 2021 01:24
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request May 12, 2021
@nchaulet @kibanamachine
[Fleet] Support dataset with multiple level like system.process.summa…
42c9004 
…ry (elastic#99852)
@kibanamachine kibanamachine mentioned this pull request May 12, 2021
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request May 12, 2021
@nchaulet @kibanamachine
[Fleet] Support dataset with multiple level like system.process.summa…
dfeb974 
…ry (elastic#99852)
@kibanamachine kibanamachine mentioned this pull request May 12, 2021
Merged
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented May 12, 2021

💚 Backport successful

Status Branch Result
✅ 7.13
✅ 7.x

The backport PRs will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request May 12, 2021
@kibanamachine @nchaulet
[Fleet] Support dataset with multiple level like system.process.summa…
5f7893b 
…ry (#99852) (#99874)

Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
kibanamachine added a commit that referenced this pull request May 12, 2021
@kibanamachine @nchaulet
[Fleet] Support dataset with multiple level like system.process.summa…
0937fe6 
…ry (#99852) (#99875)

Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
@fearful-symmetry fearful-symmetry mentioned this pull request May 13, 2021
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@jen-huang jen-huang jen-huang approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@nchaulet nchaulet

Labels

auto-backport Deprecated - use backport:version if exact versions are needed bug Fixes for quality problems that affect the customer experience release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.13.0 v7.14.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Metrics UI] Summary of process information is different between Metricbeat and Agent.

4 participants

@nchaulet @elasticmachine @kibanamachine @jen-huang