[libbeat] New decode xml wineventlog processor by marc-gr · Pull Request #25115 · elastic/beats

marc-gr · 2021-04-15T16:06:38Z

What does this PR do?

Draft of the changes required to move wineventlog decoding to a new processor

From the discussion at #25109 to see which approach we prefer.

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2021-04-15T18:03:59Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25115 updated
Start Time: 2021-04-20T07:26:49.899+0000
Duration: 60 min 33 sec
Commit: cfcb4f1

Test stats 🧪

Test	Results
Failed	0
Passed	47145
Skipped	5134
Total	52279

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	47145
Skipped	5134
Total	52279

leehinman

still testing but found 2 things

libbeat/processors/decode_xml_wineventlog/processor.go

elasticmachine · 2021-04-19T10:33:04Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

leehinman

LGTM. Thank You.

mergify · 2021-04-19T22:40:35Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b new_decode_xml_wineventlog_processor upstream/new_decode_xml_wineventlog_processor
git merge upstream/master
git push upstream new_decode_xml_wineventlog_processor

andrewkroh · 2021-04-19T23:12:56Z

libbeat/processors/decode_xml_wineventlog/docs/decode_xml_wineventlog.asciidoc

I think it would be more clear to explicitly state that this writes keys outside of target_field.

andrewkroh · 2021-04-19T23:15:26Z

libbeat/processors/decode_xml_wineventlog/docs/decode_xml_wineventlog.asciidoc

I don't think this will be useful for event log XML data. I can't think of a field that has enough uniqueness on its own. I'd would probably use a fingerprint processor to combine a few fields like record_id, channel, computer_name, timestamp if I wanted a unique _id.

…-github-pr-comment-template * upstream/master: [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) Add svc to agent k8s clusterRole (elastic#25146) Add awsfargate module to collect container logs from Amazon ECS on Fargate (elastic#25041) [Filebeat][Cisco ASA] log enhancement and performance (elastic#24744) Watch kubernetes namespaces for autodiscover metadata for pods (elastic#25117) Cyberark Privileged Access Security module (elastic#24803) [Elastic Agent] Log the container command output with LOGS_PATH (elastic#25150) Fix for tests after `device...` field has been removed (elastic#25141) [Ingest Manager] Restart process on output change (elastic#24907) Set --insecure in container when FLEET_SERVER_ENABLE and FLEET_INSECURE set. (elastic#25137) [filebeat] Update documentation / changelog / beta warnings for the syslog input (elastic#25047) Add support for ignore_inactive in filestream input (elastic#25036) Fix bug with annotations dedot config on k8s not used (elastic#25111)

* Move enrich raw functionality to common package * Enrich Raw fields when possible in decode_xml * Add ECS mappings when decoding wineventlog xml * Add decode_xml_wineventlog processor * Add missing fields to config checks * Change event.code type * Fix PR number in changelog * Fix test * Remove document_id and make docs more clear (cherry picked from commit 8cf8f51)

* Move enrich raw functionality to common package * Enrich Raw fields when possible in decode_xml * Add ECS mappings when decoding wineventlog xml * Add decode_xml_wineventlog processor * Add missing fields to config checks * Change event.code type * Fix PR number in changelog * Fix test * Remove document_id and make docs more clear (cherry picked from commit 8cf8f51) Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

…ng-versions-stack * upstream/master: (28 commits) Add support for parsers in filestream input (elastic#24763) Skip flaky test TestFilestreamTruncate (elastic#25218) backport: Add 7.13 branch (elastic#25189) Update decode_json_fields.asciidoc (elastic#25056) [Elastic Agent] Fix status and inspect command to work inside running container (elastic#25204) Check native environment before starting (elastic#25186) Change event.code and winlog.event_id type (elastic#25176) [Ingest Manager] Proxy processes/elastic-agent to stats (elastic#25193) Update mergify backporting to 7.x and 7.13 (elastic#25196) [Heartbeat]: ensure synthetics version co* [Heartbeat]: ensure synthetics version compatability for suites * address review and fix notice * fix lowercase struct * fix version conflict and rebase * update go.* stuff to master * fix notice.txt * move validate inside sourcempatability for suites (elastic#24777) [Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. (elastic#25101) Update replace.asciidoc (elastic#25055) Fix nil panic when overwriting metadata (elastic#24741) [Filebeat] Add Malware Bazaar to Threat Intel Module (elastic#24570) Fix k8s svc selectors mapping (elastic#25169) [Ingest Manager] Make agent retry values for bootstraping configurable (elastic#25163) [Metricbeat] Remove elasticsearc.index.created from the SM code (elastic#25113) [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) ...

marc-gr added enhancement Team:Security-External Integrations labels Apr 15, 2021

marc-gr requested a review from leehinman April 15, 2021 16:06

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 15, 2021

leehinman requested changes Apr 15, 2021

View reviewed changes

libbeat/processors/decode_xml_wineventlog/processor.go Outdated Show resolved Hide resolved

libbeat/processors/decode_xml_wineventlog/processor.go Outdated Show resolved Hide resolved

marc-gr marked this pull request as ready for review April 19, 2021 10:33

marc-gr force-pushed the new_decode_xml_wineventlog_processor branch from c739801 to 3f8b423 Compare April 19, 2021 15:56

marc-gr mentioned this pull request Apr 19, 2021

[libbeat] Decode xml enhance wineventlog #25109

Closed

5 tasks

marc-gr requested a review from leehinman April 19, 2021 15:58

leehinman approved these changes Apr 19, 2021

View reviewed changes

marc-gr added the backport-v7.13.0 Automated backport with mergify label Apr 19, 2021

andrewkroh approved these changes Apr 19, 2021

View reviewed changes

marc-gr force-pushed the new_decode_xml_wineventlog_processor branch from ab53093 to 7eafa15 Compare April 20, 2021 06:13

marc-gr added 9 commits April 20, 2021 09:25

Move enrich raw functionality to common package

ea2a84f

Enrich Raw fields when possible in decode_xml

9987efe

Add ECS mappings when decoding wineventlog xml

16155f3

Add decode_xml_wineventlog processor

0c2fca8

Add missing fields to config checks

61aac68

Change event.code type

38c6a32

Fix PR number in changelog

d6ec2fd

Fix test

50209e1

Remove document_id and make docs more clear

cfcb4f1

marc-gr force-pushed the new_decode_xml_wineventlog_processor branch from 3fefb41 to cfcb4f1 Compare April 20, 2021 07:25

adriansr approved these changes Apr 20, 2021

View reviewed changes

marc-gr merged commit 8cf8f51 into elastic:master Apr 20, 2021

marc-gr deleted the new_decode_xml_wineventlog_processor branch April 20, 2021 08:31

mergify bot mentioned this pull request Apr 20, 2021

[libbeat] New decode xml wineventlog processor (backport #25115) #25165

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libbeat] New decode xml wineventlog processor#25115

[libbeat] New decode xml wineventlog processor#25115
marc-gr merged 9 commits intoelastic:masterfrom
marc-gr:new_decode_xml_wineventlog_processor

marc-gr commented Apr 15, 2021

Uh oh!

elasticmachine commented Apr 15, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Uh oh!

leehinman left a comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Apr 19, 2021

Uh oh!

leehinman left a comment

Uh oh!

mergify bot commented Apr 19, 2021

Uh oh!

andrewkroh Apr 19, 2021

Uh oh!

andrewkroh Apr 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

marc-gr commented Apr 15, 2021

What does this PR do?

Why is it important?

Checklist

Uh oh!

elasticmachine commented Apr 15, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

leehinman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Apr 19, 2021

Uh oh!

leehinman left a comment

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Apr 19, 2021

Uh oh!

andrewkroh Apr 19, 2021

Choose a reason for hiding this comment

Uh oh!

andrewkroh Apr 19, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented Apr 15, 2021 •

edited by jenkins-beats-ci bot

Loading