Skip to content

[System] Add RFC 5424 support to Auth datastream#8103

Merged
kcreddy merged 8 commits intoelastic:mainfrom
kcreddy:system_auth_5424
Oct 26, 2023
Merged

[System] Add RFC 5424 support to Auth datastream#8103
kcreddy merged 8 commits intoelastic:mainfrom
kcreddy:system_auth_5424

Conversation

@kcreddy
Copy link
Copy Markdown
Contributor

@kcreddy kcreddy commented Oct 5, 2023 •
edited
Loading

Proposed commit message

  • Adds support for RFC 5424 to system.auth datastream using syslog filebeat processor.
    • For users directly utilising the assets such as ingest pipeline without agent/beat, changes are made to existing GROK patterns to support RFC 5424
  • Add tags, improve error.message, and add pipeline_error to event.kind upon on_failure.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

elastic-package stack down && elastic-package build && elastic-package stack up --version=8.10.3 -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v

--- Test results for package: system - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ system  │ auth        │ pipeline  │ test-5424.log            │ PASS   │  10.007542ms │
│ system  │ auth        │ pipeline  │ test-auth-rhel79.log     │ PASS   │   3.304583ms │
│ system  │ auth        │ pipeline  │ test-auth-ubuntu1204.log │ PASS   │  21.605917ms │
│ system  │ auth        │ pipeline  │ test-auth.log            │ PASS   │   4.547708ms │
│ system  │ auth        │ pipeline  │ test-multiline.log       │ PASS   │   3.199833ms │
│ system  │ auth        │ pipeline  │ test-secure-rhel7.log    │ PASS   │   3.846833ms │
│ system  │ auth        │ pipeline  │ test-timestamp.log       │ PASS   │   2.669458ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: system - END   ---
Done

elastic-package stack down && elastic-package build && elastic-package stack up --version=8.10.3 -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v

--- Test results for package: system - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ system  │ auth        │ system    │ default   │ PASS   │ 37.581683125s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: system - END   ---
Done

Related issues

Screenshots

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 5, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-26T10:20:46.359+0000

  • Duration: 17 min 15 sec

Test stats 🧪

Test Results
Failed 0
Passed 154
Skipped 0
Total 154

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 5, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (3/3) 💚
Files 100.0% (4/4) 💚
Classes 100.0% (4/4) 💚
Methods 65.476% (55/84) 👎 -19.246
Lines 98.415% (2981/3029) 👍 1.801
Conditionals 100.0% (0/0) 💚

kcreddy
kcreddy commented Oct 23, 2023
View reviewed changes
@kcreddy kcreddy marked this pull request as ready for review October 23, 2023 10:23
@kcreddy kcreddy requested review from a team as code owners October 23, 2023 10:23
@kcreddy kcreddy self-assigned this Oct 23, 2023
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 23, 2023

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6
efd6 reviewed Oct 23, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused by some of this.

@kcreddy kcreddy requested a review from efd6 October 24, 2023 05:02
muthu-mps
muthu-mps approved these changes Oct 26, 2023
View reviewed changes
packages/system/data_stream/auth/fields/fields.yml
ignore_above: 1024
name: version
type: keyword
# To be moved to ecs.yml after ECS version upgrade
Copy link
Copy Markdown
Contributor

@muthu-mps muthu-mps Oct 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have created an issue to keep track of updating the ECS version and reference the additional sys.log fields from there.

@kcreddy kcreddy merged commit 8236557 into elastic:main Oct 26, 2023
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 26, 2023

Package system - 1.47.0 containing this change is available at https://epr.elastic.co/search?package=system

@kcreddy kcreddy deleted the system_auth_5424 branch October 27, 2023 07:45
@rdner rdner mentioned this pull request Mar 27, 2024
Open
@jvalente-salemstate jvalente-salemstate mentioned this pull request Apr 14, 2024
Closed
@cmacknz cmacknz mentioned this pull request Jul 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 approved these changes

@muthu-mps muthu-mps muthu-mps approved these changes

Assignees

@kcreddy kcreddy

Labels

enhancement New feature or request Integration:system System

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kcreddy @elasticmachine @efd6 @muthu-mps