packages/windows/data_stream/powershell_operations: don't split tokens on hyphen by efd6 · Pull Request #1931 · elastic/integrations

efd6 · 2021-10-17T23:45:43Z

What does this PR do?

The change replaces the simple tokenizer with a custom tokenizer that splits on word boundaries that do not include hyphen.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Author's Checklist

Confirm that this generates the correct output template.

How to test this PR locally

Related issues

Fixes Hyphens ignored in script_block_text field - Powershell Logs #1776.

Screenshots

elasticmachine · 2021-10-17T23:49:53Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-11-16T00:52:11.679+0000
Duration: 19 min 33 sec
Commit: d8cd9c1

Test stats 🧪

Test	Results
Failed	0
Passed	126
Skipped	0
Total	126

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

mtojek · 2021-10-18T07:25:34Z

packages/windows/data_stream/powershell_operational/fields/fields.yml

I guess we don't support analyzers defined in fields files:

�[36mkibana_1 |�[0m {"type":"log","@timestamp":"2021-10-18T00:00:08+00:00","tags":["error","plugins","fleet"],"pid":1228,"message":"Error: Error installing windows 1.2.4: illegal_argument_exception: [illegal_argument_exception] Reason: composable template [logs-windows.powershell_operational] template after composition with component templates [logs-windows.powershell_operational@custom, .fleet_component_template-1] is invalid\n at ensureInstalledPackage (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.js:193:11)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async Promise.all (index 0)\n at PackagePolicyService.create (/usr/share/kibana/x-pack/plugins/fleet/server/services/package_policy.js:133:33)\n at createPackagePolicyHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/package_policy/handlers.js:109:27)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"}

Did you try using the ingest pipeline to approach this problem?

~~I haven't, but I think a keyword analyzer here and lowercase and split on this pattern in the ingest should work.~~ Nope.

@ruflin Do you think we need to support analyzers or is there any workaround available?

Analyzers do appear to work given that we allow settings declared the data stream manifest.yml (source: https://github.com/elastic/package-spec/blob/a0687c0dc7a9da3fc540bdc7c5df2d7d84ae6713/versions/1/data_stream/manifest.spec.yml#L174-L176).

I tested this and it passed the system test.

diff --git a/packages/windows/data_stream/powershell_operational/fields/fields.yml b/packages/windows/data_stream/powershell_operational/fields/fields.yml index 2049ba44..ae35dff3 100644 --- a/packages/windows/data_stream/powershell_operational/fields/fields.yml +++ b/packages/windows/data_stream/powershell_operational/fields/fields.yml @@ -105,10 +105,7 @@ example: "50d2dbda-7361-4926-a94d-d9eadfdb43fa" - name: script_block_text type: text - analyzer: - powershell: - type: pattern - pattern: "[\\W&&[^-]]+" + analyzer: powershell_script_analyzer description: > Text of the executed script block. diff --git a/packages/windows/data_stream/powershell_operational/manifest.yml b/packages/windows/data_stream/powershell_operational/manifest.yml index 08b887b3..8eca400c 100644 --- a/packages/windows/data_stream/powershell_operational/manifest.yml +++ b/packages/windows/data_stream/powershell_operational/manifest.yml @@ -1,5 +1,13 @@ type: logs title: Windows Powershell/Operational logs +elasticsearch: + index_template: + settings: + analysis: + analyzer: + powershell_script_analyzer: + type: pattern + pattern: '[\W&&[^-]]+' streams: - input: winlog template_path: winlog.yml.hbs

The logs-windows.powershell_operational@settings component template is created as

Spec issue created: elastic/package-spec#238

elasticmachine · 2021-10-20T06:32:50Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2021-10-20T17:32:53Z

I think we would want the same analyzer applied to these as well. Right?

integrations/packages/windows/data_stream/forwarded/fields/fields.yml

Line 145 in d7e83a6

- name: script_block_text

integrations/packages/windows/data_stream/powershell/fields/fields.yml

Line 106 in d7e83a6

- name: script_block_text

efd6 · 2021-10-21T01:01:41Z

The other question is whether the search analyzer should also be provided.

andrewkroh · 2021-10-26T14:07:54Z

The other question is whether the search analyzer should also be provided.

I think it does need a search_analyzer. Otherwise a search term like Invoke-WebRequest will get split up and not match what has been indexed.

GET /.ds-logs-windows.powershell_operational-ep-2021.10.26-000001/_analyze
{
  "field" : "powershell.file.script_block_text",
  "text" : "(Invoke-WebRequest -Uri \"https://aka.ms/pscore6-docs\").Links.Href"
}

{
  "tokens" : [
    {
      "token" : "invoke-webrequest",
      "start_offset" : 1,
      "end_offset" : 18,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "-uri",
      "start_offset" : 19,
      "end_offset" : 23,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : "https",
      "start_offset" : 25,
      "end_offset" : 30,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "aka",
      "start_offset" : 33,
      "end_offset" : 36,
      "type" : "word",
      "position" : 3
    },
    {
      "token" : "ms",
      "start_offset" : 37,
      "end_offset" : 39,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : "pscore6-docs",
      "start_offset" : 40,
      "end_offset" : 52,
      "type" : "word",
      "position" : 5
    },
    {
      "token" : "links",
      "start_offset" : 55,
      "end_offset" : 60,
      "type" : "word",
      "position" : 6
    },
    {
      "token" : "href",
      "start_offset" : 61,
      "end_offset" : 65,
      "type" : "word",
      "position" : 7
    }
  ]
}

GET /_analyze
{
  "text" : "Invoke-WebRequest"
}

{
  "tokens" : [
    {
      "token" : "invoke",
      "start_offset" : 0,
      "end_offset" : 6,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "webrequest",
      "start_offset" : 7,
      "end_offset" : 17,
      "type" : "<ALPHANUM>",
      "position" : 1
    }
  ]
}

…s on hyphen Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

efd6 added bug Something isn't working, use only for issues Team:Security-External Integrations 7.16-candidate labels Oct 17, 2021

efd6 force-pushed the windows/powershell branch from a59d507 to 78c1cf6 Compare October 17, 2021 23:46

efd6 mentioned this pull request Oct 18, 2021

x-pack/winlogbeat/module/powershell: don't split tokens on hyphen elastic/beats#28483

Merged

7 tasks

mtojek self-requested a review October 18, 2021 07:19

mtojek reviewed Oct 18, 2021

View reviewed changes

efd6 force-pushed the windows/powershell branch from 78c1cf6 to 3bcf0f7 Compare October 20, 2021 05:55

efd6 marked this pull request as ready for review October 20, 2021 06:32

mtojek mentioned this pull request Oct 20, 2021

Support Elasticsearch analyzers elastic/package-spec#238

Open

efd6 force-pushed the windows/powershell branch from 3bcf0f7 to d4fe240 Compare October 21, 2021 00:58

efd6 force-pushed the windows/powershell branch from d4fe240 to 2d4d50d Compare October 26, 2021 21:50

efd6 force-pushed the windows/powershell branch from 2d4d50d to 64ed53a Compare November 9, 2021 03:25

efd6 mentioned this pull request Nov 11, 2021

libbeat: add support for defining analyzers in-line in fields.yml files elastic/beats#28926

Merged

5 tasks

packages/windows/data_stream/powershell_operations: don't split token…

d8cd9c1

…s on hyphen Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

efd6 force-pushed the windows/powershell branch from 64ed53a to d8cd9c1 Compare November 16, 2021 00:51

efd6 requested a review from andrewkroh November 16, 2021 00:53

andrewkroh approved these changes Nov 16, 2021

View reviewed changes

efd6 merged commit 4390268 into elastic:master Nov 16, 2021

efd6 deleted the windows/powershell branch November 16, 2021 01:51

w0rk3r mentioned this pull request Mar 2, 2023

[Rule Tuning] Potential Antimalware Scan Interface Bypass via PowerShell elastic/detection-rules#2614

Merged

w0rk3r mentioned this pull request Jun 28, 2023

[Rule Tuning] [BUG] Revert PowerShell Query modifications from #2823 elastic/detection-rules#2889

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

packages/windows/data_stream/powershell_operations: don't split tokens on hyphen#1931

packages/windows/data_stream/powershell_operations: don't split tokens on hyphen#1931
efd6 merged 1 commit intoelastic:masterfrom
efd6:windows/powershell

efd6 commented Oct 17, 2021

Uh oh!

elasticmachine commented Oct 17, 2021 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

mtojek Oct 18, 2021

Uh oh!

efd6 Oct 18, 2021 •

edited

Loading

Uh oh!

mtojek Oct 18, 2021

Uh oh!

andrewkroh Oct 19, 2021 •

edited

Loading

Uh oh!

mtojek Oct 20, 2021

Uh oh!

elasticmachine commented Oct 20, 2021

Uh oh!

andrewkroh commented Oct 20, 2021

Uh oh!

efd6 commented Oct 21, 2021

Uh oh!

andrewkroh commented Oct 26, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

efd6 commented Oct 17, 2021

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Oct 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

mtojek Oct 18, 2021

Choose a reason for hiding this comment

Uh oh!

efd6 Oct 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtojek Oct 18, 2021

Choose a reason for hiding this comment

Uh oh!

andrewkroh Oct 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtojek Oct 20, 2021

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 20, 2021

Uh oh!

andrewkroh commented Oct 20, 2021

Uh oh!

efd6 commented Oct 21, 2021

Uh oh!

andrewkroh commented Oct 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elasticmachine commented Oct 17, 2021 •

edited

Loading

efd6 Oct 18, 2021 •

edited

Loading

andrewkroh Oct 19, 2021 •

edited

Loading

andrewkroh commented Oct 26, 2021 •

edited

Loading