Add more filtering options to journald input by kvch · Pull Request #29294 · elastic/beats

kvch · 2021-12-06T15:42:18Z

What does this PR do?

This PR adds support for unit, transports and syslog_identifiers options for filtering.

This PR also introduces a breaking change to include_matches option. From now on it does not accept a list of expressions. Now both conjunction (AND) and disjunctions (OR) are supported when applying matches to journals.

Collecting entries of two different units:

- type: journald
  include_matches.or:
  - equals:
    - _SYSTEMD_UNIT=my_unit
    - _SYSTEMD_UNIT=my_other_unit

Collecting entries using syslog transport for a unit

- type: journald
  include_matches.and:
  - equals:
    - _SYSTEMD_UNIT=my_unit
    - _TRANSPORT=syslog

Although the configuration lets you write complex expressions, systemd does not provide full logical expression support.

Why is it important?

When this change merged, journald input can be marked either beta or GA. Furthermore, now it provides similar filtering capabilities as the good old community Journalbeat did.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Supersedes #10985

elasticmachine · 2021-12-06T15:43:04Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

mergify · 2021-12-06T15:43:13Z

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-12-06T15:58:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-01-04T11:01:34.855+0000
Duration: 101 min 39 sec
Commit: 8dae41b

Test stats 🧪

Test	Results
Failed	0
Passed	9642
Skipped	1275
Total	10917

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Co-authored-by: Tiago Queiroz <contato@tiago.eti.br>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

…journald-filtering

andrewkroh · 2021-12-16T00:03:48Z

WDYT about doing this in a way to allows the old configs to continue to work:

I'm wondering if there's a way to implement it without breaking earlier configs though. Like if a users had include_matches: [_TRANSPORT=audit] we just re-write it to include_matches.equals: [_TRANSPORT=audit] and log a deprecation warning.

Like if include_matches is a []string then automatically convert it to the proper format.

mergify · 2021-12-16T15:27:11Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-filebeat-journald-filtering upstream/feature-filebeat-journald-filtering
git merge upstream/master
git push upstream feature-filebeat-journald-filtering

…journald-filtering

belimawr

Oops, I selected the wrong option

mergify · 2021-12-20T07:17:27Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-filebeat-journald-filtering upstream/feature-filebeat-journald-filtering
git merge upstream/master
git push upstream feature-filebeat-journald-filtering

belimawr

It looks great, just need to solve the merge conflicts.

belimawr · 2021-12-29T09:24:34Z

+	coreDumpMsgID  = MustBuildMatcher("message_id=fc2e22bc6ee647b6b90729ab34a250b1")
+	journaldUID    = MustBuildMatcher("journald.uid=0")
+	journaldPID    = MustBuildMatcher("journald.pid=1")


I like the comment you added to ApplyUnitMatchers, however I think it could also be added here, so they don't look like magic strings to the reader.

Anyway, it's not a blocker ;)

…journald-filtering

dikshachauhan-qasource · 2022-02-15T12:18:16Z

Hi @kvch

We need details around Acceptance Criteria here, so that we can validate same.

As of now, I Have gone through Filebeat.yml file and could not find any updates or reference entries regarding journald been included under Filebeat from 8.0. Could you please help us with more details here.

Thanks
QAS

kvch added 2 commits December 6, 2021 15:48

Add new filtering options: unit, identifier, kernel

1d50fa7

more

323bbfe

kvch mentioned this pull request Dec 6, 2021

Initialize filtering functionality #10985

Closed

1 task

botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 6, 2021

kvch added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team v8.1.0 labels Dec 6, 2021

botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 6, 2021

mergify Bot added the backport-skip Skip notification from the automated backport with mergify label Dec 6, 2021

kvch added backport-v8.1.0 Automated backport with mergify needs_team Indicates that the issue/PR needs a Team:* label and removed backport-skip Skip notification from the automated backport with mergify labels Dec 6, 2021

botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 6, 2021

kvch added 10 commits December 6, 2021 19:08

update go mod

8d22abf

minor refactoring

a507798

update notice and reference configuration

aa07290

rm bad import

a62c93b

add minimal tests

14d7f7a

add more build tags

3775743

minor refactoring && more tests

c7e2645

rm unnecessary lines

5a9b4d2

mod docs

88111b2

add changelog entry

b4e442c

kvch requested review from andrewkroh and belimawr December 8, 2021 13:03

kvch changed the title ~~Add journal field matching to journald input~~ Add more filtering options to journald input Dec 8, 2021

andrewkroh added breaking change release-note:breaking The content should be included as a breaking change labels Dec 8, 2021

kvch and others added 7 commits December 13, 2021 13:15

Update filebeat/docs/inputs/input-journald.asciidoc

9a1de8d

Co-authored-by: Tiago Queiroz <contato@tiago.eti.br>

Update filebeat/input/journald/pkg/journalfield/matcher.go

a38e2af

Co-authored-by: Tiago Queiroz <contato@tiago.eti.br>

Update filebeat/input/journald/pkg/journalfield/matcher.go

d79137e

Co-authored-by: Tiago Queiroz <contato@tiago.eti.br>

address review notes i.

49ef567

address review notes ii.

0f636db

Add link to fields reference

495cf93

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

missing follow up

b702358

This comment has been minimized.

Sign in to view

kvch added 4 commits December 14, 2021 16:31

follow up in tests

75b6665

update docs

a446b2a

Merge remote-tracking branch 'upstream/master' into feature-filebeat-…

6651a4c

…journald-filtering

fix docs

be2a5bf

andrewkroh reviewed Dec 16, 2021

View reviewed changes

Comment thread filebeat/docs/inputs/input-journald.asciidoc

Comment thread filebeat/input/journald/pkg/journalfield/matcher.go

Comment thread filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl

kvch added 2 commits December 16, 2021 17:07

add list of transports to docs

89e811c

more address

5d79aff

kvch requested a review from belimawr December 17, 2021 09:49

Merge remote-tracking branch 'upstream/master' into feature-filebeat-…

646e976

…journald-filtering

belimawr approved these changes Dec 17, 2021

View reviewed changes

Comment thread CHANGELOG.next.asciidoc

Comment thread filebeat/input/journald/pkg/journalfield/matcher.go Outdated

belimawr requested changes Dec 17, 2021

View reviewed changes

rm changelog && add more docs

b81047c

belimawr approved these changes Dec 29, 2021

View reviewed changes

kvch added 2 commits January 4, 2022 11:50

Merge remote-tracking branch 'upstream/master' into feature-filebeat-…

dbf24b5

…journald-filtering

add more comments to matchers

8dae41b

kvch merged commit fc7b8fc into elastic:master Jan 4, 2022

This was referenced Jan 26, 2022

[8.0] iptables system test: found 0 hits in logs-iptables.log-ep data stream elastic/integrations#2602

Closed

[Filebeat] journald: Make include_matches backwards compatible with 7.x config #30032

Merged

Conversation

kvch commented Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Related issues

Uh oh!

elasticmachine commented Dec 6, 2021

Uh oh!

mergify Bot commented Dec 6, 2021

Uh oh!

elasticmachine commented Dec 6, 2021 • edited by jenkins-beats-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

andrewkroh commented Dec 16, 2021

Uh oh!

mergify Bot commented Dec 16, 2021

Uh oh!

Uh oh!

Uh oh!

belimawr left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Dec 20, 2021

Uh oh!

belimawr left a comment

Choose a reason for hiding this comment

Uh oh!

belimawr Dec 29, 2021

Choose a reason for hiding this comment

Uh oh!

dikshachauhan-qasource commented Feb 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kvch commented Dec 6, 2021 •

edited

Loading

elasticmachine commented Dec 6, 2021 •

edited by jenkins-beats-ci Bot

Loading