Skip to content

[system.auth] Split grok parsing, add tags/processors/preserve_original_event#3705

Merged
andrewkroh merged 7 commits intoelastic:mainfrom
andrewkroh:system/auth-performance
Jul 20, 2022
Merged

[system.auth] Split grok parsing, add tags/processors/preserve_original_event#3705
andrewkroh merged 7 commits intoelastic:mainfrom
andrewkroh:system/auth-performance

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh commented Jul 13, 2022 •
edited
Loading

What does this PR do?

This adds config options for tags, processors, and preserve original event to the system.auth data stream.

It improves the pipeline by splitting the main grok processor into two operations. First it groks the header common to all messages. Then the second grok extracts information from a specific set of messages. Previously all of the patterns in the single grok shared the same header so the the header was being parsed multiple times while trying to find a matching pattern. Additionally the patterns where not anchored so those were added.

I added a convert processor to check the source.ip for validity.

I added an if condition to the script processor to prevent cannot access method/field [event] from a null def reference errors that were being ignored with ignore_failure: true.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshots

Screen Shot 2022-07-14 at 12 29 48

Before:

logs-system auth-1 17 0-metrics

After:

logs-system auth-1 18 0-metrics

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Jul 13, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-07-19T19:06:05.857+0000

  • Duration: 17 min 9 sec

Test stats 🧪

Test Results
Failed 0
Passed 271
Skipped 0
Total 271

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@andrewkroh
Fix ignored failures in the pipeline
2ecaa9b 
The script failed with "cannot access method/field [event] from a null def reference".

And the geoip failed when the source.ip was missing.
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Jul 14, 2022 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (3/3) 💚
Files 100.0% (4/4) 💚 3.036
Classes 100.0% (4/4) 💚 3.036
Methods 59.74% (46/77) 👎 -29.365
Lines 98.788% (2689/2722) 👍 8.428
Conditionals 100.0% (0/0) 💚

@andrewkroh andrewkroh added Team:Security-External Integrations Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane] labels Jul 14, 2022
@andrewkroh andrewkroh requested a review from a team July 14, 2022 16:45
@andrewkroh andrewkroh marked this pull request as ready for review July 14, 2022 16:46
@andrewkroh andrewkroh requested a review from a team as a code owner July 14, 2022 16:46
@andrewkroh andrewkroh requested review from cmacknz and faec July 14, 2022 16:46
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Jul 14, 2022

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Jul 14, 2022

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@andrewkroh andrewkroh mentioned this pull request Jul 14, 2022
Merged
6 tasks
efd6
efd6 reviewed Jul 14, 2022
View reviewed changes
@andrewkroh andrewkroh requested a review from efd6 July 19, 2022 20:15
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Jul 19, 2022
@andrewkroh
system.auth - sync pipeline with Fleet integration
8fadb43 
Sync the pipeline for the system.auth dataset with the Fleet integration
from elastic/integrations#3705.

This removes the event.type authentication_failed and authentication_success
values which are not allowed as per ECS. You can use event.category: authentication
and event.outcome: success/failure to query instead.
@andrewkroh andrewkroh merged commit 12e2963 into elastic:main Jul 20, 2022
andrewkroh added a commit to elastic/beats that referenced this pull request Jul 20, 2022
@andrewkroh
system.auth - sync pipeline with Fleet integration (#32360)
475dd7e 
Sync the pipeline for the system.auth dataset with the Fleet integration
from elastic/integrations#3705.

This removes the event.type authentication_failed and authentication_success
values which are not allowed as per ECS. You can use event.category: authentication
and event.outcome: success/failure to query instead.
mergify bot pushed a commit to elastic/beats that referenced this pull request Jul 20, 2022
@andrewkroh @mergify
system.auth - sync pipeline with Fleet integration (#32360)
186ce93 
Sync the pipeline for the system.auth dataset with the Fleet integration
from elastic/integrations#3705.

This removes the event.type authentication_failed and authentication_success
values which are not allowed as per ECS. You can use event.category: authentication
and event.outcome: success/failure to query instead.

(cherry picked from commit 475dd7e)
andrewkroh added a commit to elastic/beats that referenced this pull request Jul 26, 2022
@mergify @andrewkroh
[7.17](backport #32360) system.auth - sync pipeline with Fleet integr…
03ded87 
…ation (#32422)

* system.auth - sync pipeline with Fleet integration (#32360)

Sync the pipeline for the system.auth dataset with the Fleet integration
from elastic/integrations#3705.

This removes the event.type authentication_failed and authentication_success
values which are not allowed as per ECS. You can use event.category: authentication
and event.outcome: success/failure to query instead.

(cherry picked from commit 475dd7e)

* Update go-ucfg to v0.8.6

The module tests were failing because the empty tags array was being discarded
resulting in the tags value being treated as `nil`. And this was not being accepted as
a valid type for the inList function.

The cause was elastic/go-ucfg#188.

* Fix tests affected by go-ucfg upgrade

Backport fix from 0022ea4.

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
@ppf2
Copy link
Copy Markdown

ppf2 commented Jul 27, 2022

@andrewkroh Is this change getting back port to 7.17 branch for future Beats 7.17.x releases, or will it only be in 8.3.x/8.4+? Thx!

@andrewkroh
Copy link
Copy Markdown
Member Author

andrewkroh commented Jul 27, 2022

The related Filebeat changes are

That covers 8.4 and 7.17.

@andrewkroh andrewkroh mentioned this pull request Aug 25, 2022
Merged
chrisberkhout pushed a commit to elastic/beats that referenced this pull request Jun 1, 2023
@andrewkroh @chrisberkhout
system.auth - sync pipeline with Fleet integration (#32360)
66f33d7 
Sync the pipeline for the system.auth dataset with the Fleet integration
from elastic/integrations#3705.

This removes the event.type authentication_failed and authentication_success
values which are not allowed as per ECS. You can use event.category: authentication
and event.outcome: success/failure to query instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 approved these changes

@cmacknz cmacknz Awaiting requested review from cmacknz cmacknz was automatically assigned from elastic/elastic-agent-data-plane

@faec faec Awaiting requested review from faec faec was automatically assigned from elastic/elastic-agent-data-plane

Assignees

No one assigned

Labels

Integration:system System Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@andrewkroh @elasticmachine @ppf2 @efd6