Skip to content

Add KQL functionality in the find function of the saved objects#41136

Merged
XavierM merged 14 commits intoelastic:masterfrom
XavierM:add-kql-to-savedObject
Oct 2, 2019
Merged

Add KQL functionality in the find function of the saved objects#41136
XavierM merged 14 commits intoelastic:masterfrom
XavierM:add-kql-to-savedObject

Conversation

@XavierM
Copy link
Copy Markdown
Contributor

@XavierM XavierM commented Jul 15, 2019

Summary

The SIEM app is migrating to using SavedObjects which also means migrating all their queries. Since these are written in KQL rewriting them as Simple Query Strings will be time-consuming and error-prone. In addition, Simple Query String doesn't support range queries.

Since KQL is designed as a simple filter language it doesn't replace searching with Simple Query Strings in Saved Objects but rather can be used in addition to constructing more powerful queries.

Dev Docs

SavedObjectsClient.find now supports filtering using a KQL string, with the caveat that if you filter with an attribute from your type saved object, it should look like this: savedObjectType.attributes.name: "SayMyName". However, If you use a direct attribute of a saved object like updatedAt, you will have to define your filter like this: savedObjectType.updatedAt > 2018-12-22.

savedObjectsClient.find({
      type: 'savedObjectType',
      sortField: '@timestamp',
      sortOrder: 'desc',
      search: '',
      searchFields:'',
      fields: ['id', 'name', '@created', '@timestamp'],
      filter:
        'savedObjectType.attributes.name: "SayMyName" and savedObjectType.updatedAt > 2018-12-22'
    });

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

For maintainers

@rudolf
Copy link
Copy Markdown
Contributor

rudolf commented Jul 17, 2019

Mike Cote brought up another use case that could be solved by having KQL: https://github.com/elastic/kibana/pull/39829/files#diff-5598f367eaa6aef5c6bbe4654d764a9bR130

@XavierM XavierM force-pushed the add-kql-to-savedObject branch from d6e65c6 to 3a51743 Compare July 18, 2019 10:19
@XavierM XavierM added review release_note:skip Skip the PR/issue when compiling release notes and removed discuss labels Jul 18, 2019
@rudolf rudolf added the Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// label Jul 18, 2019
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/kibana-platform

Copy link
Copy Markdown
Contributor

@rudolf rudolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some initial comments

@XavierM XavierM force-pushed the add-kql-to-savedObject branch from 3a51743 to 38bae52 Compare July 18, 2019 13:00
@rudolf
Copy link
Copy Markdown
Contributor

rudolf commented Jul 19, 2019

Edit: added a 3rd option.

@elastic/kibana-platform In order to evaluate KQL expressions the KQL parser requires a StaticIndexPattern which has the following shape:

{
  fields: [
    name: string;
    type: string;
    aggregatable: boolean;
    searchable: boolean;
  ];
  title: string;
}

There are two ways to collect this information:

  1. IndexPatternsService src/legacy/server/index_patterns.js which uses ES field capabilities api
  2. Construct it from the SavedObject mappings
  3. Use the ES field capabilities api directly inside core/saved_objects without moving the whole IndexPatternsService to Core.

If we use (1) it means we'll have to move this service to Core, I haven't seen this discussed, but I assume the current plan is for it to live in the data plugin. (2) is a bit more complex and it means maintaining a field type -> {searchable, aggregatable} mapping instead of leveraging the ES API which is guaranteed to be up to date.

The way @XavierM implemented it now the SavedObjects Repository only requires the compiled static index patterns, so it's easy to swap around how we construct these when we move everything to Core.

@XavierM XavierM force-pushed the add-kql-to-savedObject branch from 172d56a to c8af3d0 Compare July 19, 2019 13:44
Copy link
Copy Markdown
Contributor

@rudolf rudolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Github stopped letting me add comments 🚫 so submitting what I have.

@XavierM XavierM force-pushed the add-kql-to-savedObject branch from 05305c6 to ccb5da5 Compare July 19, 2019 17:05
@elasticmachine

This comment has been minimized.

@XavierM XavierM force-pushed the add-kql-to-savedObject branch from ccb5da5 to 78bf464 Compare July 22, 2019 07:05
@elastic elastic deleted a comment from elasticmachine Jul 22, 2019
@elastic elastic deleted a comment from elasticmachine Jul 22, 2019
@elastic elastic deleted a comment from elasticmachine Jul 22, 2019
@elastic elastic deleted a comment from elasticmachine Jul 22, 2019
@XavierM XavierM force-pushed the add-kql-to-savedObject branch from 6f9ef53 to 9ac6de7 Compare October 2, 2019 11:08
@elastic elastic deleted a comment from elasticmachine Oct 2, 2019
@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

@XavierM XavierM requested a review from lukasolson October 2, 2019 18:00
@XavierM XavierM force-pushed the add-kql-to-savedObject branch from 9ac6de7 to e72f7f5 Compare October 2, 2019 18:29
@elasticmachine

This comment has been minimized.

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Copy Markdown
Contributor

💔 Build Failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature:Saved Objects release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. review Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// v7.5.0 v8.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants