wintest: new package to provide support for winlogbeat ingest node pipeline testing by efd6 · Pull Request #31833 · elastic/beats

efd6 · 2022-06-06T11:39:50Z

What does this PR do?

This adds infrastructure for testing ingest pipelines in winlogbeat.

Why is it important?

Winlogbeat's ingest pipelines are currently untested and provisional testing shows that there will be failures when testing is instituted. Tests will make it possible to fix these problems.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Testing needs to be run with -p 1, otherwise there is contention for the container's network port if package test globbing is used.
~~The linter complaint can be completely ignored; it is due to staticcheck knowing that a t.Skip is a terminating statement and so that later uses are not ever executed.~~ Tests are now passing, so no Skip is being done.
The tests are currently in unittest, but it could be argued they should be in integ test. I'm not wedded to either, but it was easier to work without build constraints. I don't believe that the CI is currently running the tests here, and I'm not sure how to wire them in. So help there would be appreciated.

Failing in Powershell:

--- FAIL: TestPowerShellIngest (25.85s)
    --- FAIL: TestPowerShellIngest/800.evtx.golden.json (0.00s)
        testing.go:158: unexpected ingest error for event 2: unexpected pipeline error: [scripting] Regular expression considered too many characters, pattern: [^(.+)\\((.+)\\)\\:\\s*(.+)?$], limit factor: [6], char limit: [1386], count: [1387], wrapped: [NonTerminatingError(Import-LocalizedData): \"Cannot find the W...], this limit can be changed by changed by the [script.painless.regex.limit-factor] setting
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestPowerShellIngest/800.evtx.golden.json
            	Messages:   	Key '_temp.details' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestPowerShellIngest/800.evtx.golden.json
            	Messages:   	Key '_temp.user_parts' found in event is not documented.
        testing.go:158: unexpected ingest error for event 3: unexpected pipeline error: [scripting] Regular expression considered too many characters, pattern: [^(.+)\\((.+)\\)\\:\\s*(.+)?$], limit factor: [6], char limit: [1470], count: [1471], wrapped: [ParameterBinding(Out-Default): name=\"InputObject\"; value=\"Can...], this limit can be changed by changed by the [script.painless.regex.limit-factor] setting
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestPowerShellIngest/800.evtx.golden.json
            	Messages:   	Key '_temp.details' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestPowerShellIngest/800.evtx.golden.json
            	Messages:   	Key '_temp.user_parts' found in event is not documented.
FAIL

Failing in Security:

--- FAIL: TestSecurityIngest (26.16s)
    --- FAIL: TestSecurityIngest/4706_WindowsSrv2016.evtx.golden.json (0.00s)
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4706_WindowsSrv2016.evtx.golden.json
            	Messages:   	Key 'winlog.trustAttribute' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4706_WindowsSrv2016.evtx.golden.json
            	Messages:   	Key 'winlog.trustDirection' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4706_WindowsSrv2016.evtx.golden.json
            	Messages:   	Key 'winlog.trustType' found in event is not documented.
    --- FAIL: TestSecurityIngest/4716_WindowsSrv2016.evtx.golden.json (0.00s)
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4716_WindowsSrv2016.evtx.golden.json
            	Messages:   	Key 'winlog.trustDirection' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4716_WindowsSrv2016.evtx.golden.json
            	Messages:   	Key 'winlog.trustType' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4716_WindowsSrv2016.evtx.golden.json
            	Messages:   	Key 'winlog.trustAttribute' found in event is not documented.
    --- FAIL: TestSecurityIngest/4741.evtx.golden.json (0.00s)
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4741.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.domain' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4741.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.id' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4741.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.name' found in event is not documented.
    --- FAIL: TestSecurityIngest/4742.evtx.golden.json (0.00s)
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4742.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.name' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4742.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.id' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4742.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.domain' found in event is not documented.
    --- FAIL: TestSecurityIngest/4743.evtx.golden.json (0.00s)
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4743.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.id' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4743.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.name' found in event is not documented.
        testing.go:165: 
            	Error Trace:	testing.go:328
            	            				testing.go:165
            	Error:      	Field not documented
            	Test:       	TestSecurityIngest/4743.evtx.golden.json
            	Messages:   	Key 'winlog.computerObject.domain' found in event is not documented.
    --- FAIL: TestSecurityIngest/4908_WindowsSrv2016.evtx.golden.json (0.00s)
        testing.go:150: unexpected error: unexpected response status for simulate: 400 Bad Request (400): elasticsearch error (type=illegal_argument_exception): unexpected value type [class [Ljava.lang.String;]
            Root cause:
            [
              {
                "type": "illegal_argument_exception",
                "reason": "unexpected value type [class [Ljava.lang.String;]",
                "position": {
                  "offset": 0,
                  "start": 0,
                  "end": 0
                }
              }
            ]
    --- FAIL: TestSecurityIngest/security-windows2012_4674.evtx.golden.json (0.00s)
        testing.go:158: unexpected ingest error for event 0: unexpected pipeline error: Processor "script" with tag "Decode message table" in pipeline "" failed with message "For input string: \"%%1538\n\t\t\t\t%%1542\n\t\t\t\t\""
    --- FAIL: TestSecurityIngest/security-windows2012_4768.evtx.golden.json (0.00s)
        testing.go:197: Expected and actual are different:
            --- Expected
            +++ Actual
            @@ -63,4 +63,4 @@
                     "Forwardable",
            +        "Renewable",
                     "Renewable-ok",
            -        "Renewable",
                     "Name-canonicalize"
    --- FAIL: TestSecurityIngest/security-windows2012_4771.evtx.golden.json (0.00s)
        testing.go:197: Expected and actual are different:
            --- Expected
            +++ Actual
            @@ -58,4 +58,4 @@
                     "Forwardable",
            +        "Renewable",
                     "Renewable-ok",
            -        "Renewable",
                     "Name-canonicalize"
FAIL

Note that the final two failures in Security look to be due to instability in the ordering of addition to an array and this may need a fix in the comparison code in the module test.

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2022-06-06T12:06:54Z

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2022-06-08T00:11:35Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh

I'm part way through the changes. I'll finish up tomorrow. Looks good.

x-pack/winlogbeat/module/testing.go

efd6 · 2022-06-09T06:30:11Z

x-pack/winlogbeat/module/security/test/security_ingest_test.go

The inconsistency in this could be avoided by using bit operations in the painless rather than the map iteration that is currently done.

I expect, but have not observed, that the same issue will be present in the AccessMaskDescription processing.

efd6 · 2022-06-09T06:40:47Z

x-pack/winlogbeat/module/security/ingest/security.yml

The fix would be to iterate over the word size of the flags, e.g. for the AccessMaskDescription case

for (def b = 0; b < 32; b++) { long accessFlag = 1L << b; if (accessMask&accessFlag == accessFlag) {

and an appropriate access to the parameter map — maybe via a hex string format of the accessFlag if an integer map key is not possible.

The equivalent change can be made for TicketOptionsDescription at line 876 in this file.

mergify · 2022-06-09T12:12:57Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b wintest/docker upstream/wintest/docker
git merge upstream/main
git push upstream wintest/docker

efd6 · 2022-06-14T08:15:31Z

AFAICS the E2E failures are unrelated.

cmacknz · 2022-06-14T13:12:36Z

AFAICS the E2E failures are unrelated.

Yes the E2E tests have been unstable for a few weeks, I am looking into disabling the flakiest ones when triggered from beats but not from agent.

x-pack/winlogbeat/module/pipeline.go

x-pack/winlogbeat/module/wintest/testdata/README

x-pack/winlogbeat/magefile.go

x-pack/winlogbeat/module/testing.go

…peline testing Includes wiring in module tests, but skipped for powershell and security due to failing tests.

…800 parameter detail processing

…ocessing

This guarantees that flag description lists will be in a canonical order, and and should be less work.

efd6 · 2022-06-23T00:04:05Z

Rebased with the hope that the tests pass.

efd6 · 2022-06-23T02:49:56Z

E2E failure is unrelated, End-To-End Tests / fleet_ debian_10_arm64_fleet_mode_agent:

[2022-06-23T02:24:58.636Z] [INFO] withAPMEnv: is enabled.
Cannot get property 'shell_type' on null object

…peline testing (#31833) - powershell: fix regexp constraints in event 800 parameter detail processing - security: fix documentation - security: fix sidlist processing - security: fix access mask and access list processing

efd6 added enhancement Winlogbeat Team:Security-External Integrations backport-skip Skip notification from the automated backport with mergify 8.4-candidate labels Jun 6, 2022

efd6 self-assigned this Jun 6, 2022

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 6, 2022

efd6 force-pushed the wintest/docker branch from fdf49a6 to 8bf9afd Compare June 6, 2022 11:59

efd6 force-pushed the wintest/docker branch 5 times, most recently from ebe7550 to 67c624d Compare June 8, 2022 00:07

efd6 marked this pull request as ready for review June 8, 2022 00:11

efd6 requested review from a team as code owners June 8, 2022 00:11

efd6 requested review from cmacknz and rdner and removed request for a team June 8, 2022 00:11

efd6 force-pushed the wintest/docker branch from 67c624d to b4d95fd Compare June 8, 2022 00:14

efd6 mentioned this pull request Jun 8, 2022

winlogbeat: disentangle test data from x-pack/winlogbeat #31858

Merged

6 tasks

efd6 force-pushed the wintest/docker branch 6 times, most recently from a52f1ac to 8dfeec6 Compare June 8, 2022 09:33

This was referenced Jun 9, 2022

windows/powershell: regexp considered too many characters for processing 800 command invocation details elastic/integrations#3494

Closed

windows/powershell: fix regexp constraints in event 800 parameter detail processing elastic/integrations#3495

Merged

andrewkroh reviewed Jun 9, 2022

View reviewed changes

x-pack/winlogbeat/module/testing.go Outdated Show resolved Hide resolved

efd6 force-pushed the wintest/docker branch 3 times, most recently from 231bcbd to dd6ae23 Compare June 9, 2022 06:11

efd6 commented Jun 9, 2022

View reviewed changes

efd6 force-pushed the wintest/docker branch 2 times, most recently from 08fd54d to 8050192 Compare June 14, 2022 04:53

rdner reviewed Jun 15, 2022

View reviewed changes

x-pack/winlogbeat/module/pipeline.go Outdated Show resolved Hide resolved

x-pack/winlogbeat/module/wintest/testdata/README Outdated Show resolved Hide resolved

efd6 force-pushed the wintest/docker branch from 8050192 to 82205dd Compare June 15, 2022 10:11

andrewkroh approved these changes Jun 22, 2022

View reviewed changes

x-pack/winlogbeat/magefile.go Outdated Show resolved Hide resolved

x-pack/winlogbeat/module/testing.go Outdated Show resolved Hide resolved

efd6 added 8 commits June 23, 2022 09:31

wintest: new package to provide support for winlogbeat ingest node pi…

a0b7444

…peline testing Includes wiring in module tests, but skipped for powershell and security due to failing tests.

x-pack/winlogbeat/module/powershell: fix regexp constraints in event …

587f084

…800 parameter detail processing

x-pack/winlogbeat/module/security: fix documentation

6178221

x-pack/winlogbeat/module/security: fix sidlist processing

40dd37c

x-pack/winlogbeat/module/security: fix access mask and access list pr…

a5b7090

…ocessing

add changelog line for security processor fixes

3216433

x-pack/winlogbeat/module/security: use bitshifts to examine flag states

24963e7

This guarantees that flag description lists will be in a canonical order, and and should be less work.

address pr comments

72b5324

efd6 force-pushed the wintest/docker branch from 82205dd to 72b5324 Compare June 23, 2022 00:01

andrewkroh approved these changes Jun 23, 2022

View reviewed changes

efd6 merged commit a401b8e into elastic:main Jun 23, 2022

efd6 mentioned this pull request Jun 24, 2022

[Winlogbeat] add tests for ingest node pipelines #30406

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wintest: new package to provide support for winlogbeat ingest node pipeline testing#31833

wintest: new package to provide support for winlogbeat ingest node pipeline testing#31833
efd6 merged 8 commits intoelastic:mainfrom
efd6:wintest/docker

efd6 commented Jun 6, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Jun 6, 2022 •

edited by jenkins-beats-ci bot

Loading

Uh oh!

elasticmachine commented Jun 8, 2022

Uh oh!

andrewkroh left a comment

Uh oh!

Uh oh!

efd6 Jun 9, 2022

Uh oh!

efd6 Jun 9, 2022 •

edited

Loading

Uh oh!

mergify bot commented Jun 9, 2022

Uh oh!

efd6 commented Jun 14, 2022

Uh oh!

cmacknz commented Jun 14, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Jun 23, 2022

Uh oh!

efd6 commented Jun 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

efd6 commented Jun 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Jun 6, 2022 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Flaky test report

🤖 GitHub comments

Uh oh!

elasticmachine commented Jun 8, 2022

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efd6 Jun 9, 2022

Choose a reason for hiding this comment

Uh oh!

efd6 Jun 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Jun 9, 2022

Uh oh!

efd6 commented Jun 14, 2022

Uh oh!

cmacknz commented Jun 14, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Jun 23, 2022

Uh oh!

efd6 commented Jun 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

efd6 commented Jun 6, 2022 •

edited

Loading

elasticmachine commented Jun 6, 2022 •

edited by jenkins-beats-ci bot

Loading

efd6 Jun 9, 2022 •

edited

Loading