[Ingest Manager] Manually build Fleet kuery with Node arguments by nchaulet · Pull Request #76589 · elastic/kibana

nchaulet · 2020-09-03T01:24:22Z

Summary

This PR #75693 introduced a way to manually build filter instead of parsing the KQL string to improve perfomance. While improving a lot the performance there is still room for improvment if we use buildNodeWithArgumentNodes.

Benchmarking the potential improvment

  const suite = new Benchmark.Suite();

  suite
    .add('parse KQL', function () {
      esKuery.fromKueryExpression(
        'not fleet-agent-actions.attributes.sent_at: * and fleet-agent-actions.attributes.agent_id:1234567'
      );
    })
    .add('buildNode', function () {
      esKuery.nodeTypes.function.buildNode('and', [
        esKuery.nodeTypes.function.buildNode(
          'not',
          esKuery.nodeTypes.function.buildNode('is', 'fleet-agent-actions.attributes.sent_at', '*')
        ),
        esKuery.nodeTypes.function.buildNode(
          'is',
          'fleet-agent-actions.attributes.agent_id',
          '1234567'
        ),
      ]);
    })
    .add('buildNodeWithArgumentNodes', function () {
      nodeTypes.function.buildNodeWithArgumentNodes('and', [
        nodeTypes.function.buildNode(
          'not',
          nodeTypes.function.buildNodeWithArgumentNodes('is', [
            nodeTypes.literal.buildNode(`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.sent_at`),
            nodeTypes.wildcard.buildNode(nodeTypes.wildcard.wildcardSymbol),
            nodeTypes.literal.buildNode(false),
          ])
        ),
        nodeTypes.function.buildNodeWithArgumentNodes('is', [
          nodeTypes.literal.buildNode(`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.agent_id`),
          nodeTypes.literal.buildNode('123456'),
          nodeTypes.literal.buildNode(false),
        ]),
      ]);
    })
    // add listeners
    .on('cycle', function (event) {
      console.log(String(event.target));
    })
    .on('complete', function () {
      console.log('Fastest is ' + this.filter('fastest').map('name'));
    })
    // run async
    .run({ async: true, minSamples: 200 });

Give the following result

parse KQL x 1,005 ops/sec ±4.58% (81 runs sampled)
buildNode x 6,143 ops/sec ±4.75% (84 runs sampled)
buildNodeWithArgumentNodes x 905,728 ops/sec ±5.77% (84 runs sampled)
Fastest is buildNodeWithArgumentNodes

elasticmachine · 2020-09-03T01:24:25Z

Pinging @elastic/ingest-management (Team:Ingest Management)

kobelb · 2020-09-03T13:36:22Z

Very nice! I did not know this was an option. @lukasolson, you're the KQL expert, does this make sense to you?

lukasolson · 2020-09-03T17:45:28Z

Hmm, if this performance improvement is really necessary, this is fine, but I'd generally like to avoid using buildNodeWithArgumentNodes outside of the grammar.peg file, since it leads to worse readability.

kobelb · 2020-09-03T18:22:16Z

@lukasolson perhaps we should address this by providing a better interface for building these KQL queries which hide the intricacies? I believe you mentioned considering implementing this before, is there an existing GitHub issue for that?

nchaulet · 2020-09-08T19:13:51Z

@elasticmachine merge upstream

nchaulet · 2020-09-09T15:27:21Z

@lukasolson Do you think we can move forward with this PR?

lukasolson · 2020-09-09T16:48:37Z

Yes, it's fine. I've opened #77085 to track building a better programmatic API.

nchaulet · 2020-09-09T17:40:20Z

@lukasolson thanks for the issue, a simple API like the one you put in the issue will be amazing :)

There is one change to the WildcardType in this PR if you have time for a review

nchaulet · 2020-09-14T14:52:21Z

@elasticmachine merge upstream

kibanamachine · 2020-09-14T16:24:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 7067bba

Build metrics

✅ unchanged

History

💚 Build #73206 succeeded ee2dd65
💚 Build #72120 succeeded 7ed80d5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…tic#76589)

…) (#77413)

* master: (25 commits) [Security Solution] Add unit tests for Network search strategy (elastic#77416) [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing (elastic#77040) [Ingest Manager] Add route for package installation by upload (elastic#77044) [APM-UI][E2E] filter PRs from the uptime GH team (elastic#77359) [APM] Remove useLocation and some minor route improvements (elastic#76343) [Enterprise Search] Update enterpriseSearchRequestHandler to manage range of errors + add handleAPIErrors helper (elastic#77258) [SECURITY_SOLUTION] Task/hostname policy response ux updates (elastic#76444) Move remaining uses of serviceName away from urlParams (elastic#77248) [Lens] Move configuration popover to flyout (elastic#76046) [Ingest Manager] Manually build Fleet kuery with Node arguments (elastic#76589) skip flaky suite (elastic#59975) Neutral-naming in reporting plugin (elastic#77371) [Enterprise Search] Add UserIcon styles (elastic#77385) [RUM Dashboard] Added loading state to visitor breakdown pie charts (elastic#77201) [Ingest Manager] Fix polling for new agent action (elastic#77339) Remote cluster - Functional UI test to change the superuser to a test_user with limited role (elastic#77212) Stacked headers and navigational search (elastic#72331) [ML] DF Analytics creation wizard: Fixing field loading race condition (elastic#77326) [Monitoring] Handle no mappings found for sort and collapse fields (elastic#77099) Add Lens to Recently Accessed (elastic#77249) ...

* master: (293 commits) Fix tsvb filter ration for table (elastic#77272) [Security Solution] Add unit tests for Network search strategy (elastic#77416) [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing (elastic#77040) [Ingest Manager] Add route for package installation by upload (elastic#77044) [APM-UI][E2E] filter PRs from the uptime GH team (elastic#77359) [APM] Remove useLocation and some minor route improvements (elastic#76343) [Enterprise Search] Update enterpriseSearchRequestHandler to manage range of errors + add handleAPIErrors helper (elastic#77258) [SECURITY_SOLUTION] Task/hostname policy response ux updates (elastic#76444) Move remaining uses of serviceName away from urlParams (elastic#77248) [Lens] Move configuration popover to flyout (elastic#76046) [Ingest Manager] Manually build Fleet kuery with Node arguments (elastic#76589) skip flaky suite (elastic#59975) Neutral-naming in reporting plugin (elastic#77371) [Enterprise Search] Add UserIcon styles (elastic#77385) [RUM Dashboard] Added loading state to visitor breakdown pie charts (elastic#77201) [Ingest Manager] Fix polling for new agent action (elastic#77339) Remote cluster - Functional UI test to change the superuser to a test_user with limited role (elastic#77212) Stacked headers and navigational search (elastic#72331) [ML] DF Analytics creation wizard: Fixing field loading race condition (elastic#77326) [Monitoring] Handle no mappings found for sort and collapse fields (elastic#77099) ...

nchaulet added v8.0.0 v7.10.0 Team:Fleet Team label for Observability Data Collection Fleet team labels Sep 3, 2020

nchaulet requested review from a team and kobelb September 3, 2020 01:24

nchaulet self-assigned this Sep 3, 2020

nchaulet added the release_note:skip Skip the PR/issue when compiling release notes label Sep 3, 2020

nchaulet force-pushed the feature-manually-build-fleet-kuery branch from d18b7a0 to a0b7c77 Compare September 3, 2020 01:34

nchaulet requested a review from a team as a code owner September 3, 2020 01:34

nchaulet mentioned this pull request Sep 3, 2020

[Ingest Manager] Improve perfomance of fetching unacknowledged actions #75892

Closed

1 task

[Ingest Manager] Manually build Fleet kuery with Node arguments

7ed80d5

nchaulet force-pushed the feature-manually-build-fleet-kuery branch from a0b7c77 to 7ed80d5 Compare September 3, 2020 01:39

kobelb approved these changes Sep 4, 2020

View reviewed changes

Merge branch 'master' into feature-manually-build-fleet-kuery

ee2dd65

lukasolson mentioned this pull request Sep 9, 2020

[KQL] Better programmatic API #77085

Closed

nchaulet requested a review from jfsiii September 9, 2020 19:01

lukasolson approved these changes Sep 9, 2020

View reviewed changes

nchaulet requested a review from jen-huang September 10, 2020 19:31

Merge branch 'master' into feature-manually-build-fleet-kuery

7067bba

neptunian approved these changes Sep 14, 2020

View reviewed changes

nchaulet merged commit 5256604 into elastic:master Sep 14, 2020

nchaulet deleted the feature-manually-build-fleet-kuery branch September 14, 2020 21:19

nchaulet added a commit to nchaulet/kibana that referenced this pull request Sep 14, 2020

[Ingest Manager] Manually build Fleet kuery with Node arguments (elas…

3c6560f

…tic#76589)

nchaulet mentioned this pull request Sep 14, 2020

[7.x] [Ingest Manager] Manually build Fleet kuery with Node arguments (#76589) #77413

Merged

nchaulet added a commit that referenced this pull request Sep 15, 2020

[Ingest Manager] Manually build Fleet kuery with Node arguments (#76589…

e88a4cd

…) (#77413)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Manually build Fleet kuery with Node arguments#76589

[Ingest Manager] Manually build Fleet kuery with Node arguments#76589
nchaulet merged 3 commits intoelastic:masterfrom
nchaulet:feature-manually-build-fleet-kuery

nchaulet commented Sep 3, 2020

Uh oh!

elasticmachine commented Sep 3, 2020

Uh oh!

kobelb commented Sep 3, 2020

Uh oh!

lukasolson commented Sep 3, 2020

Uh oh!

kobelb commented Sep 3, 2020 •

edited

Loading

Uh oh!

nchaulet commented Sep 8, 2020

Uh oh!

nchaulet commented Sep 9, 2020

Uh oh!

lukasolson commented Sep 9, 2020

Uh oh!

nchaulet commented Sep 9, 2020

Uh oh!

nchaulet commented Sep 14, 2020

Uh oh!

kibanamachine commented Sep 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

nchaulet commented Sep 3, 2020

Summary

Benchmarking the potential improvment

Uh oh!

elasticmachine commented Sep 3, 2020

Uh oh!

kobelb commented Sep 3, 2020

Uh oh!

lukasolson commented Sep 3, 2020

Uh oh!

kobelb commented Sep 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nchaulet commented Sep 8, 2020

Uh oh!

nchaulet commented Sep 9, 2020

Uh oh!

lukasolson commented Sep 9, 2020

Uh oh!

nchaulet commented Sep 9, 2020

Uh oh!

nchaulet commented Sep 14, 2020

Uh oh!

kibanamachine commented Sep 14, 2020

💚 Build Succeeded

Build metrics

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kobelb commented Sep 3, 2020 •

edited

Loading