Endpoint Telemetry: Agents Metrics + Policy Config / Response by pjhampton · Pull Request #102171 · elastic/kibana

pjhampton · 2021-06-15T09:36:18Z

Summary

This PR retrieves and transmits Endpoint agent telemetry if cluster permissions permit.
There have been auxiliary PRs / Issues opened:

Granting Kbn System read permissions to read from EP Data stream elasticsearch#74309 - Grant kibana_system user to .ds-metrics-endpoint.policy-*
Grant kbn system user permissions to read Endpoint metrics datastream elasticsearch#74340 - Grant kibana_system user to .ds-metrics-endpoint.metrics-*
Open stack infra issue for telemetry channel

We are currently sharing the telemetry with the Endpoint team. We will be making changes to the final payload.

Implementation

The implementation is not that straightforward - here is a high level of how it works

Every 24hr grab an available Kibana worker
Aggregate the endpoint metrics datastream for the last 24 hours on unique endpoint id for latest metrics document

GET /.ds-metrics-endpoint.metrics*/_search?expand_wildcards=open,hidden
{
  "size": 0,
  "query": {
    "match_all": {}
  },
  "aggs": {
    "endpoint_agents": {
      "terms": { 
        "size": 10000,
        "field": "agent.id" 
      },
      "aggs": {
        "latest_metrics": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

If there is no endpoint metrics we will assume there isn't any endpoint running and bail; otherwise
Get the full list of fleet agents
Iterate through the full list of agents, if it has a policy check it is of type endpoint, add to a cache associating the fleet agent id with the policy id
Aggregate the endpoint policy datastream for the last 24 hours on a unique endpoint id for the latest failed policy response (if exists)

GET /.ds-metrics-endpoint.policy*/_search?expand_wildcards=open,hidden
{
  "size": 0,
  "query": {
    "match": {
      "Endpoint.policy.applied.status": "failure"
    }
  },
  "aggs": {
    "policy_responses": {
      "terms": {
        "field": "Endpoint.policy.applied.id",
        "size": 10
      },
      "aggs": {
        "latest_response": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

Combine the metrics documents with the correct policy configuration and response failure (if applicable)
Send to the endpoint-meta telemetry channel

Follow up

I'm hosting a call week beginning 28/Jun re this telemetry + design.
Let me know if you want me to swing you an invite.

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2021-06-15T09:36:20Z

Pinging @elastic/security-solution (Team: SecuritySolution)

… EP metrics

pjhampton · 2021-06-17T15:17:49Z

@elasticmachine merge upstream

…ana into pjhampton/endpoint-telemetry

x-pack/plugins/security_solution/server/lib/telemetry/endpoint_task.ts

pjhampton · 2021-06-29T09:09:49Z

@elasticmachine merge upstream

Bamieh

LGTM

michaelolo24 · 2021-06-29T16:13:20Z

x-pack/plugins/security_solution/server/lib/telemetry/endpoint_task.ts

+
+    const endpointPolicyCache = new Map<string, FullAgentPolicyInput>();
+    for (const policyInfo of fleetAgents.values()) {
+      if (policyInfo.policy_id !== null && policyInfo.policy_id !== undefined) {


Just a personal preference here, but might make it a bit easier to read intent if we store the boolean logic in a variable

const shouldCachePolicy = policyInfo.policy_id !== null && policyInfo.policy_id !== undefined && !endpointPolicyCache.has(policyInfo.policy_id)

Yeah, that is slick. Thanks for the feedback!

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

michaelolo24

Looks good! Thanks for making these changes. We can work on performance improvements on a follow up PR 👍🏾

pjhampton · 2021-06-30T07:23:59Z

@elasticmachine merge upstream

kibanamachine · 2021-06-30T09:33:47Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #135288 succeeded 99ceeef
💔 Build #135259 failed 4e5804b
💚 Build #135054 succeeded cbd499d
💚 Build #134989 succeeded eb75a93
💚 Build #134546 succeeded 0781c54

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pjhampton @donaherc

…c#102171) * [PH] Initial setup for endpoint task telemetry. * Refactor / Add daily task for collecting fleet detail / policy resp / EP metrics * [PH CD] Code walkthrough. Start fetching fleet policy configs. * [PH] pass in fleet agent service rather than homebrew kuerys. * [PH] prepare to move away from legacy es client. Get fleet ep agents. * Fetch agent policy configs. * Stub ep policy responses. * Fix CI + Types. Fix dep injection. Reimagine SO client creation. * Create SO client properly * Fetch EP Policy responses. * Fetch EP Policy responses. * Remove unused import * Fetch failed policy responses from EP data stream. * Remove unused imports. * Combine failed policy responses with policy configs. * Attach fleet agent + ep agent ids * Add dedicated channel sender. Temp disable with feature flag. * Remove ublock from the failed policy response. * Fetch endpoint metrics. * Fix bad merge commit. * Get EP telemetry. * Record last execution time of endpoint task * Remove send on demand feature flag. * Simplify cache conditional. * Refactor into Promise.allSettled * Fix type error. * Bail if there is no endpoint metrics * Bump interval to 24h. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-06-30T09:37:00Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

… (#103851) * [PH] Initial setup for endpoint task telemetry. * Refactor / Add daily task for collecting fleet detail / policy resp / EP metrics * [PH CD] Code walkthrough. Start fetching fleet policy configs. * [PH] pass in fleet agent service rather than homebrew kuerys. * [PH] prepare to move away from legacy es client. Get fleet ep agents. * Fetch agent policy configs. * Stub ep policy responses. * Fix CI + Types. Fix dep injection. Reimagine SO client creation. * Create SO client properly * Fetch EP Policy responses. * Fetch EP Policy responses. * Remove unused import * Fetch failed policy responses from EP data stream. * Remove unused imports. * Combine failed policy responses with policy configs. * Attach fleet agent + ep agent ids * Add dedicated channel sender. Temp disable with feature flag. * Remove ublock from the failed policy response. * Fetch endpoint metrics. * Fix bad merge commit. * Get EP telemetry. * Record last execution time of endpoint task * Remove send on demand feature flag. * Simplify cache conditional. * Refactor into Promise.allSettled * Fix type error. * Bail if there is no endpoint metrics * Bump interval to 24h. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Pete Hampton <pjhampton@users.noreply.github.com>

…-png-pdf-report-type * 'master' of github.com:elastic/kibana: (178 commits) [test] Migrating to kbn_archiver from es_archiver - for the Maps app (elastic#103028) [Reporting] Reintroduce "ILM policy for managing reporting indices" (elastic#103850) [Security Solution][Endpoint] Allow activity log scrolling on small screens (elastic#103852) Allow zero (0) to unset unenroll_timeout field (elastic#103790) [TSVB] Metric count is depicted as `-` instead of 0 (elastic#103717) [Query] Es query/field base (elastic#103177) Remove add data button from nav (elastic#103810) Fix telemetry advanced setting style (elastic#103838) [Transform] Fix default naming and sorting fields suggestion for `top_metrics` agg (elastic#103690) [APM] use conventional error rate color for correlations (elastic#103500) Endpoint Telemetry: Agents Metrics + Policy Config / Response (elastic#102171) [Alerting] Fixed search results are not updated when search term is removed on Rules and Connectors page (elastic#103663) fix too many rernders (elastic#103672) [APM] Add “Analyze Data” button (elastic#103485) [Lens] Fix value popover spacing (elastic#103081) [TSVB] Fix TSVB is not reporting all categories of Elasticsearch error (elastic#102926) [SECURITY] Adds security links to doc link service (elastic#102676) Update dependency @elastic/charts to v31 (elastic#102078) [Security Solution][CTI] Investigation time enrichment UI (elastic#103383) Adds ECS guide to doc links service (elastic#102246) ... # Conflicts: # x-pack/plugins/reporting/public/share_context_menu/register_pdf_png_reporting.tsx

[PH] Initial setup for endpoint task telemetry.

1432e1b

pjhampton added Feature:Telemetry v8.0.0 Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.14.0 auto-backport Deprecated - use backport:version if exact versions are needed labels Jun 15, 2021

pjhampton self-assigned this Jun 15, 2021

pjhampton requested a review from a team as a code owner June 15, 2021 09:36

pjhampton marked this pull request as draft June 15, 2021 09:36

pjhampton added the release_note:skip Skip the PR/issue when compiling release notes label Jun 15, 2021

pjhampton added 2 commits June 15, 2021 11:44

Refactor / Add daily task for collecting fleet detail / policy resp /…

4fa12e3

… EP metrics

[PH CD] Code walkthrough. Start fetching fleet policy configs.

4c75e55

pjhampton assigned donaherc Jun 15, 2021

pjhampton added 6 commits June 16, 2021 12:17

[PH] pass in fleet agent service rather than homebrew kuerys.

92e8e0d

[PH] prepare to move away from legacy es client. Get fleet ep agents.

26420dc

Fetch agent policy configs.

e9a3c4c

Stub ep policy responses.

9a72714

Fix CI + Types. Fix dep injection. Reimagine SO client creation.

f2ddc22

Create SO client properly

5975ed7

Merge branch 'master' into pjhampton/endpoint-telemetry

7b9d43c

pjhampton mentioned this pull request Jun 18, 2021

Granting Kbn System read permissions to read from EP Data stream elastic/elasticsearch#74309

Merged

pjhampton added 4 commits June 18, 2021 15:27

Fetch EP Policy responses.

7b6a365

Fetch EP Policy responses.

0661f22

Merge branch 'pjhampton/endpoint-telemetry' of github.com:elastic/kib…

a2308c8

…ana into pjhampton/endpoint-telemetry

Remove unused import

699e068

pjhampton mentioned this pull request Jun 21, 2021

Grant kbn system user permissions to read Endpoint metrics datastream elastic/elasticsearch#74340

Merged

pjhampton added 2 commits June 21, 2021 14:04

Fetch failed policy responses from EP data stream.

b810618

Remove unused imports.

6932470

Bamieh reviewed Jun 28, 2021

View reviewed changes

x-pack/plugins/security_solution/server/lib/telemetry/endpoint_task.ts Show resolved Hide resolved

Record last execution time of endpoint task

ee38ae7

Merge branch 'master' into pjhampton/endpoint-telemetry

eb75a93

pjhampton mentioned this pull request Jun 29, 2021

Remove <7.14 EP Metrics from Security Solution usage collector #103632

Merged

Remove send on demand feature flag.

cbd499d

pjhampton changed the title ~~Endpoint Agent Telemetry: Fleet Config + Policy Response~~ Endpoint Telemetry: Agents Metrics + Policy Config / Response Jun 29, 2021

Bamieh approved these changes Jun 29, 2021

View reviewed changes

michaelolo24 reviewed Jun 29, 2021

View reviewed changes

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts Show resolved Hide resolved

pjhampton added 5 commits June 29, 2021 18:13

Simplify cache conditional.

464db0d

Refactor into Promise.allSettled

4e5804b

Fix type error.

d769be8

Bail if there is no endpoint metrics

d88d151

Bump interval to 24h.

99ceeef

michaelolo24 approved these changes Jun 29, 2021

View reviewed changes

pjhampton enabled auto-merge (squash) June 29, 2021 19:13

Merge branch 'master' into pjhampton/endpoint-telemetry

c4316b0

pjhampton merged commit eca1460 into master Jun 30, 2021

kibanamachine mentioned this pull request Jun 30, 2021

[7.x] Endpoint Telemetry: Agents Metrics + Policy Config / Response (#102171) #103851

Merged

pjhampton mentioned this pull request Jul 6, 2021

Update index names for endpoint Datastream permissions. elastic/elasticsearch#74959

Closed

spalger added the v7.15.0 label Jul 7, 2021

pjhampton mentioned this pull request Jul 20, 2021

Fix up undefined handling in EP Meta telem task #106269

Merged

2 tasks

pjhampton deleted the pjhampton/endpoint-telemetry branch February 15, 2022 09:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Endpoint Telemetry: Agents Metrics + Policy Config / Response#102171

Endpoint Telemetry: Agents Metrics + Policy Config / Response#102171
pjhampton merged 37 commits intomasterfrom
pjhampton/endpoint-telemetry

pjhampton commented Jun 15, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Jun 15, 2021

Uh oh!

pjhampton commented Jun 17, 2021

Uh oh!

Uh oh!

pjhampton commented Jun 29, 2021

Uh oh!

Bamieh left a comment

Uh oh!

michaelolo24 Jun 29, 2021

Uh oh!

pjhampton Jun 29, 2021

Uh oh!

Uh oh!

michaelolo24 left a comment

Uh oh!

pjhampton commented Jun 30, 2021

Uh oh!

kibanamachine commented Jun 30, 2021

Uh oh!

kibanamachine commented Jun 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

pjhampton commented Jun 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation

Follow up

Checklist

Uh oh!

elasticmachine commented Jun 15, 2021

Uh oh!

pjhampton commented Jun 17, 2021

Uh oh!

Uh oh!

pjhampton commented Jun 29, 2021

Uh oh!

Bamieh left a comment

Choose a reason for hiding this comment

Uh oh!

michaelolo24 Jun 29, 2021

Choose a reason for hiding this comment

Uh oh!

pjhampton Jun 29, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

michaelolo24 left a comment

Choose a reason for hiding this comment

Uh oh!

pjhampton commented Jun 30, 2021

Uh oh!

kibanamachine commented Jun 30, 2021

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

kibanamachine commented Jun 30, 2021

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pjhampton commented Jun 15, 2021 •

edited

Loading