[ML] Fix custom URLs processing for security app by darnautov · Pull Request #76957 · elastic/kibana

darnautov · 2020-09-08T15:53:07Z

Summary

Closes #76789

Fixes token replacement for queries with spaces around the field-value separator.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2020-09-08T15:53:10Z

Pinging @elastic/ml-ui (:ml)

x-pack/plugins/ml/public/application/util/custom_url_utils.test.ts

peteharverson · 2020-09-09T16:50:37Z

x-pack/plugins/ml/public/application/util/custom_url_utils.ts

            .split(/\sand\s/i)
            // Get property name from `influencerField:$influencerField$` string.
-            .map((v) => v.split(':')[0]);
+            .map((v) => String(v.split(/:(.+)?\$/)[0]).trim());


Tested a custom URL on the special characters data set and am seeing this error when opening a link to Dashboard:

Job config has:

"analysis_config": { "bucket_span": "15m", "detectors": [ { "detector_description": "mean(\"metric%$£&!{(]field\") partitionfield=\"at@name\"", "function": "mean", "field_name": "metric%$£&!{(]field", "partition_field_name": "at@name", "detector_index": 0 } ], "influencers": [ "at@name", "singlequote'name" ] },

And opening a URL which uses the influencers from this anomaly:

I think it is the singlequote'name field name which is causing the problem here.

Fixed in 57fa5a0

…tiry-custom-urls

peteharverson · 2020-09-14T09:39:14Z

Found another issue with the special characters data, when the field value is contains a ' quote:

Opens dashboard with this in the query bar:
at@name:"contains a !' quote" AND singlequote'name:"contains a !' quote" (with an ! going in before the quote).

custom URL I am testing is

      {
        "url_name": "Show dashboard",
        "time_range": "auto",
        "url_value": """dashboards#/view/351de820-f2bb-11ea-ab06-cb93221707e9?_a=(filters:!(),query:(language:kuery,query:'at@name:"$at@name$" and singlequote!'name:"$singlequote!'name$"'))&_g=(filters:!(),time:(from:'$earliest$',mode:absolute,to:'$latest$'))"""
      },

peteharverson · 2020-09-14T09:48:20Z

x-pack/plugins/ml/public/application/util/custom_url_utils.test.ts

+    test('return expected url for Security app', () => {
+      const urlConfig = {
+        url_name: 'Hosts Overview by process name',
+        url_value:


Probably worth adding a test for the slightly more complicated 'Host details by process name' custom URL, which uses host.name and process.name:

{ "url_name": "Host Details by process name", "url_value": "security/hosts/ml-hosts/$host.name$?_g=()&query=(query:'process.name%20:%20%22$process.name$%22',language:kuery)&timerange=(global:(linkTo:!(timeline),timerange:(from:'$earliest$',kind:absolute,to:'$latest$')),timeline:(linkTo:!(global),timerange:(from:'$earliest$',kind:absolute,to:'$latest$')))" },

added in e8b935e

darnautov · 2020-09-14T10:37:12Z

Found another issue with the special characters data, when the field value is contains a ' quote:

Opens dashboard with this in the query bar:
at@name:"contains a !' quote" AND singlequote'name:"contains a !' quote" (with an ! going in before the quote).

custom URL I am testing is
      {
        "url_name": "Show dashboard",
        "time_range": "auto",
        "url_value": """dashboards#/view/351de820-f2bb-11ea-ab06-cb93221707e9?_a=(filters:!(),query:(language:kuery,query:'at@name:"$at@name$" and singlequote!'name:"$singlequote!'name$"'))&_g=(filters:!(),time:(from:'$earliest$',mode:absolute,to:'$latest$'))"""
      },

was happening because of an extra escaping for a rison string, changed in e8b935e, added a unit test in 5b34682

peteharverson

Tested latest edits against siem_auditbeat, siem_winlogbeat and special_characters jobs and LGTM.

kibanamachine · 2020-09-14T12:37:16Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e0da44a

Build metrics

async chunks size

id	value	diff	baseline
ml	8.0MB	+3.7KB	8.0MB

History

💚 Build #74075 succeeded 5b34682
💚 Build #74039 succeeded 33fc97f
💚 Build #73158 succeeded 9047528

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

qn895 · 2020-09-14T14:56:27Z

Changes LGTM 💯

* master: (26 commits) updating datatable type (elastic#77320) [ML] Fix custom URLs processing for security app (elastic#76957) [telemetry] add schema guideline + schema_check new check for --path config (elastic#75747) [ML] Transforms: API schemas and integration tests (elastic#75164) [Mappings editor] Add support for wildcard field type (elastic#76574) [Ingest Manager] Fix flyout instruction selection (elastic#77071) [Telemetry Tools] update lodash to 4.17 (elastic#77317) [APM] Service inventory redesign (elastic#76744) Hide management sections based on cluster/index privileges (elastic#67791) [Snapshot Restore] Disable steps when form is invalid (elastic#76540) [Mappings editor] Add support for positive_score_impact to rank_feature (elastic#76824) Update apm.ts (elastic#77310) [OBS] Remove beta badge, change news feed size and add external icon to news feed link (elastic#77164) [Discover] Convert legacy sort to be compatible with multi sort (elastic#76986) [APM] API Snapshot Testing (elastic#77229) [ML] Functional tests - increase wait time for DFA start (elastic#77307) [UiActions][Drilldowns] Fix actions sorting in context menu (elastic#77162) [Drilldowns] Wire up new links to new docs (elastic#77154) Fix APM issue template [Ingest Pipelines] Drop into an empty tree (elastic#76885) ...

* [ML] fix custom urls processing for security app * [ML] improve query string parsing * [ML] remove escaping with !, adjust a unit test for security app * [ML] unit test * [ML] unit test

* master: (65 commits) [Security Solution][Resolver] Analyzed event styling (elastic#77115) filter invalid SOs from the searc hresults in Task Manager (elastic#76891) [RUM Dashboard] Visitors by region map (elastic#77135) [Security Solution][Endpoint][Admin] Task/endpoint list actions (elastic#76555) [Ingest pipelines] Forms for processors T-U (elastic#76710) updating datatable type (elastic#77320) [ML] Fix custom URLs processing for security app (elastic#76957) [telemetry] add schema guideline + schema_check new check for --path config (elastic#75747) [ML] Transforms: API schemas and integration tests (elastic#75164) [Mappings editor] Add support for wildcard field type (elastic#76574) [Ingest Manager] Fix flyout instruction selection (elastic#77071) [Telemetry Tools] update lodash to 4.17 (elastic#77317) [APM] Service inventory redesign (elastic#76744) Hide management sections based on cluster/index privileges (elastic#67791) [Snapshot Restore] Disable steps when form is invalid (elastic#76540) [Mappings editor] Add support for positive_score_impact to rank_feature (elastic#76824) Update apm.ts (elastic#77310) [OBS] Remove beta badge, change news feed size and add external icon to news feed link (elastic#77164) [Discover] Convert legacy sort to be compatible with multi sort (elastic#76986) [APM] API Snapshot Testing (elastic#77229) ...

* [ML] fix custom urls processing for security app * [ML] improve query string parsing * [ML] remove escaping with !, adjust a unit test for security app * [ML] unit test * [ML] unit test

* [ML] Fix custom URLs processing for security app (#76957) * [ML] fix custom urls processing for security app * [ML] improve query string parsing * [ML] remove escaping with !, adjust a unit test for security app * [ML] unit test * [ML] unit test * [ML] update check for kibana urls

[ML] fix custom urls processing for security app

9047528

darnautov added release_note:fix :ml v8.0.0 v7.10.0 v7.9.2 labels Sep 8, 2020

darnautov requested a review from peteharverson September 8, 2020 15:53

darnautov requested a review from a team as a code owner September 8, 2020 15:53

darnautov self-assigned this Sep 8, 2020

qn895 reviewed Sep 9, 2020

View reviewed changes

x-pack/plugins/ml/public/application/util/custom_url_utils.test.ts Outdated Show resolved Hide resolved

peteharverson reviewed Sep 9, 2020

View reviewed changes

darnautov added 2 commits September 14, 2020 10:52

[ML] improve query string parsing

57fa5a0

Merge remote-tracking branch 'upstream/master' into ML-76789-fix-secu…

33fc97f

…tiry-custom-urls

peteharverson reviewed Sep 14, 2020

View reviewed changes

darnautov added 2 commits September 14, 2020 12:16

[ML] remove escaping with !, adjust a unit test for security app

e8b935e

[ML] unit test

5b34682

darnautov requested review from peteharverson and qn895 September 14, 2020 10:37

[ML] unit test

e0da44a

peteharverson approved these changes Sep 14, 2020

View reviewed changes

qn895 approved these changes Sep 14, 2020

View reviewed changes

darnautov merged commit c5358f3 into elastic:master Sep 14, 2020

darnautov deleted the ML-76789-fix-secutiry-custom-urls branch September 14, 2020 14:59

darnautov mentioned this pull request Sep 14, 2020

[7.x] [ML] Fix custom URLs processing for security app (#76957) #77367

Merged

darnautov mentioned this pull request Sep 14, 2020

[7.9] [ML] Fix custom URLs processing for security app (#76957) #77369

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Fix custom URLs processing for security app#76957

[ML] Fix custom URLs processing for security app#76957
darnautov merged 6 commits intoelastic:masterfrom
darnautov:ML-76789-fix-secutiry-custom-urls

darnautov commented Sep 8, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Sep 8, 2020

Uh oh!

Uh oh!

peteharverson Sep 9, 2020

Uh oh!

darnautov Sep 14, 2020

Uh oh!

peteharverson commented Sep 14, 2020

Uh oh!

peteharverson Sep 14, 2020

Uh oh!

darnautov Sep 14, 2020

Uh oh!

darnautov commented Sep 14, 2020

Uh oh!

peteharverson left a comment

Uh oh!

kibanamachine commented Sep 14, 2020

Uh oh!

qn895 commented Sep 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

darnautov commented Sep 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

elasticmachine commented Sep 8, 2020

Uh oh!

Uh oh!

peteharverson Sep 9, 2020

Choose a reason for hiding this comment

Uh oh!

darnautov Sep 14, 2020

Choose a reason for hiding this comment

Uh oh!

peteharverson commented Sep 14, 2020

Uh oh!

peteharverson Sep 14, 2020

Choose a reason for hiding this comment

Uh oh!

darnautov Sep 14, 2020

Choose a reason for hiding this comment

Uh oh!

darnautov commented Sep 14, 2020

Uh oh!

peteharverson left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Sep 14, 2020

💚 Build Succeeded

Build metrics

async chunks size

History

Uh oh!

qn895 commented Sep 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

darnautov commented Sep 8, 2020 •

edited

Loading