journalctl: add --exclude-identifier option by zhmylove · Pull Request #23086 · systemd/systemd

zhmylove · 2022-04-15T00:07:38Z

Exclude syslog identifiers

There is -t (--identifier=STRING) journalctl option which allows to specify SYSLOG_IDENTIFIER in a handy manner. This option could be used several times, each of them appends new layer of OR condition. A lot of journalctl(1) users are using it to show log entries related to only particular syslog identifiers. There are a lot of situations when it's required not to select a particular syslog identifiers, but to exclude some of them leaving all the others visible. There is a real problem in implementing this using Match* functionality built into journals as it match any part of the object by hash, while this could be effectively achieved during the journal entries output. This PR introduces -T (--exclude=STRING) option which implements this functionality for output_short (output_verbose, output_json, and others are intentionally left intact so far).
Example:

# ./build/journalctl |head -5
Apr 15 02:05:40 rc1 systemd-logind[364]: Removed session c11.
Apr 15 02:06:01 rc1 CRON[23073]: pam_unix(cron:session): session opened for user root by (uid=0)
Apr 15 02:06:01 rc1 CRON[23074]: (root) CMD (/root/redis/balancer.pl)
Apr 15 02:06:01 rc1 CRON[23073]: pam_unix(cron:session): session closed for user root
Apr 15 02:06:39 rc1 su[23076]: (to root) korg on pts/0
# ./build/journalctl -T CRON |head -5
Apr 15 02:05:40 rc1 systemd-logind[364]: Removed session c11.
Apr 15 02:06:39 rc1 su[23076]: (to root) korg on pts/0
Apr 15 02:06:39 rc1 su[23076]: pam_unix(su-l:session): session opened for user root by korg(uid=1000)
Apr 15 02:06:39 rc1 systemd-logind[364]: New session c12 of user root.
Apr 15 02:06:39 rc1 systemd[1]: Started Session c12 of user root.
# ./build/journalctl -T CRON -T su |head -5
Apr 15 02:05:40 rc1 systemd-logind[364]: Removed session c11.
Apr 15 02:06:39 rc1 systemd-logind[364]: New session c12 of user root.
Apr 15 02:06:39 rc1 systemd[1]: Started Session c12 of user root.
Apr 15 02:06:47 rc1 systemd[1]: session-c12.scope: Succeeded.
Apr 15 02:06:47 rc1 systemd-logind[364]: Session c12 logged out. Waiting for processes to exit.

src/basic/string-util.c

bluca

manpage update is missing. Also would be good to have a test/unit test for this, there should be already tests for the journal. Finally, rebase and squash fixup commits, so that history is clean. Would be good to have the two new options added as separate commits, since they are not related.

src/shared/logs-show.c

src/journal/journalctl.c

src/basic/string-util.c

.gitignore

src/journal/journalctl.c

zhmylove · 2022-04-15T14:35:12Z

@bluca many thanks for your review!

Would be good to have the two new options added as separate commits, since they are not related.

Should I send them as different PRs or just 2 separate commits under one PR?

I'll work on man/tests later on.

bluca · 2022-04-15T14:37:31Z

@bluca many thanks for your review!

Would be good to have the two new options added as separate commits, since they are not related.

Should I send them as different PRs or just 2 separate commits under one PR?

I'll work on man/tests later on.

same PR is fine by me

zhmylove · 2022-04-15T15:58:22Z

@bluca I've re-ordered changes into 3 separate commits and added manpage update.
Still looking for a way to add unit tests. So far I can't find any of them for journalctl.

bluca · 2022-04-15T16:00:51Z

@bluca I've re-ordered changes into 3 separate commits and added manpage update. Still looking for a way to add unit tests. So far I can't find any of them for journalctl.

There are tests for the journal in test/units/testsuite-04.sh

zhmylove · 2022-04-16T08:34:49Z

@bluca thank you for pointing me out on them.

I've added unit test for --exclude and I belive we're done with this part.

I'd like to discuss with you the test for --strip. Usually we use systemd-cat to add log messages into the log. Unfortunately it's not able of adding multiline messages as it splits stdin by newlines:

# systemd-cat -t "$ID" sh -c 'echo line1 ;echo line2'
# journalctl -q -b -t "$ID"
Apr 16 11:41:57 rc1 q[29341]: line1
Apr 16 11:41:57 rc1 q[29341]: line2

We can't rely on systemd-journal-remote as HAVE_MICROHTTPD could be false, like on my system.
Neither can we rely on logger(1) which could be missing on several systems. For example, on Debian-based it's part of bsdutils package, not the util-linux, and I believe it's not installed by default.
What will be the best way to create a multiline message in journal from the testsuite?

As python >= 3.5 is build-requirement for systemd we can try to utilize it like:

# python -c 'import syslog; syslog.openlog("qqq"); syslog.syslog("line1\nline2")'
# journalctl -q -b -t qqq
Apr 16 12:04:34 rc1 qqq[29513]: line1
                                line2

But I'm not sure it'll be available inside the testsuite. Should we add python to BASICTOOLS of test/test-functions?

Thanks!

bluca · 2022-04-16T09:11:12Z

@bluca thank you for pointing me out on them.

I've added unit test for --exclude and I belive we're done with this part.

I'd like to discuss with you the test for --strip. Usually we use systemd-cat to add log messages into the log. Unfortunately it's not able of adding multiline messages as it splits stdin by newlines:
# systemd-cat -t "$ID" sh -c 'echo line1 ;echo line2'
# journalctl -q -b -t "$ID"
Apr 16 11:41:57 rc1 q[29341]: line1
Apr 16 11:41:57 rc1 q[29341]: line2
We can't rely on systemd-journal-remote as HAVE_MICROHTTPD could be false, like on my system. Neither can we rely on logger(1) which could be missing on several systems. For example, on Debian-based it's part of bsdutils package, not the util-linux, and I believe it's not installed by default. What will be the best way to create a multiline message in journal from the testsuite?

As python >= 3.5 is build-requirement for systemd we can try to utilize it like:
# python -c 'import syslog; syslog.openlog("qqq"); syslog.syslog("line1\nline2")'
# journalctl -q -b -t qqq
Apr 16 12:04:34 rc1 qqq[29513]: line1
                                line2
But I'm not sure it'll be available inside the testsuite. Should we add python to BASICTOOLS of test/test-functions?

Thanks!

Use logger and simply check if it's available with command -v, and skip the test if not - I'll make sure it's installed in the CI

zhmylove · 2022-04-16T09:54:36Z

@bluca added test for --strip. Thank you!

bluca · 2022-04-16T10:01:53Z

@bluca added test for --strip. Thank you!

You need to add a test_append_files in test/TEST-04-JOURNAL/test.sh that installs logger if it's available - see test/TEST-43-PRIVATEUSER-UNPRIV/test.sh as an example

zhmylove · 2022-04-16T17:24:00Z

@bluca this time included. Thanks!

bluca · 2022-04-16T17:42:19Z

@bluca this time included. Thanks!

The test is now failing on s390x, you can fetch the logs from https://autopkgtest.ubuntu.com/results/autopkgtest-focal-upstream-systemd-ci-systemd-ci/focal/s390x/s/systemd-upstream/20220416_143415_22e40@/artifacts.tar.gz

bluca · 2022-04-16T17:43:32Z

here too: https://jenkins-systemd.apps.ocp.ci.centos.org/job/upstream-vagrant-archlinux-sanitizers/11170/console

src/basic/string-util.h

src/basic/string-util.c

src/journal/journalctl.c

shell-completion/bash/journalctl

src/journal/journalctl.c

src/libsystemd/sd-journal/sd-journal.c

src/shared/logs-show.c

zhmylove · 2024-01-04T12:34:04Z

@YHNdnzj

Hmm, this is not yet resolved?

Already resolved.

Also, how does this relate to #12592 ?

The idea looks similar, people do really want this functionality for a long time.
But that PR is probably abandoned by the author for last three years.
There were several of them, I think.

@bluca Thanks a lot for your approval!

@yuwata Thank you for the comments, everything is fixed now. Some CI failed due to GitHub API rate limit, nothing related to code :(

yuwata

Currently, the new option only works with output type which uses output_short().
I think such feature should be implemented in sd-journal side.
But, this is a good start point. Let's take the current way, and extend the feature later.

src/journal/journalctl.c

src/libsystemd/sd-journal/journal-internal.h

yuwata

LGTM. Thank you, and sorry for late review process.

The logger utility is now used in TEST-04-JOURNAL. See systemd/systemd#23086

github-actions bot added the journal label Apr 15, 2022

pyih reviewed Apr 15, 2022

View reviewed changes

src/basic/string-util.c Outdated Show resolved Hide resolved

bluca requested changes Apr 15, 2022

View reviewed changes

bluca added the reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks label Apr 15, 2022

zhmylove force-pushed the exclude_and_strip branch 2 times, most recently from 6e08d96 to aad68b2 Compare April 15, 2022 15:53

bluca removed the reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks label Apr 15, 2022

zhmylove force-pushed the exclude_and_strip branch from aad68b2 to f01fb3a Compare April 16, 2022 08:22

zhmylove force-pushed the exclude_and_strip branch from f01fb3a to aef1caf Compare April 16, 2022 09:53

zhmylove force-pushed the exclude_and_strip branch 2 times, most recently from e4ebadd to eb6a37c Compare April 16, 2022 13:24

bluca added the ci-fails/needs-rework 🔥 Please rework this, the CI noticed an issue with the PR label Apr 16, 2022

zhmylove force-pushed the exclude_and_strip branch 2 times, most recently from e27669a to f117805 Compare April 17, 2022 12:59

yuwata requested changes Apr 17, 2022

View reviewed changes

yuwata added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks and removed ci-fails/needs-rework 🔥 Please rework this, the CI noticed an issue with the PR labels Apr 17, 2022

zhmylove force-pushed the exclude_and_strip branch 2 times, most recently from 77c6f18 to 34e4493 Compare April 17, 2022 14:58

zhmylove force-pushed the exclude_and_strip branch from bf663b8 to 13490dd Compare January 3, 2024 16:29

bluca approved these changes Jan 3, 2024

View reviewed changes

bluca added good-to-merge/waiting-for-ci 👍 PR is good to merge, but CI hasn't passed at time of review. Please merge if you see CI has passed and removed documentation util-lib tests good-to-merge/with-minor-suggestions labels Jan 3, 2024

bluca changed the title ~~journalctl: add --exclude option~~ journalctl: add --exclude-identifier option Jan 3, 2024

github-actions bot added documentation util-lib tests labels Jan 3, 2024

yuwata requested changes Jan 3, 2024

View reviewed changes

yuwata added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks and removed good-to-merge/waiting-for-ci 👍 PR is good to merge, but CI hasn't passed at time of review. Please merge if you see CI has passed labels Jan 3, 2024

zhmylove force-pushed the exclude_and_strip branch 2 times, most recently from bcf09c3 to b53b1aa Compare January 4, 2024 12:17

github-actions bot added please-review PR is ready for (re-)review by a maintainer and removed reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels Jan 4, 2024

yuwata requested changes Jan 4, 2024

View reviewed changes

src/journal/journalctl.c Outdated Show resolved Hide resolved

src/libsystemd/sd-journal/journal-internal.h Outdated Show resolved Hide resolved

yuwata added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks and removed please-review PR is ready for (re-)review by a maintainer labels Jan 4, 2024

zhmylove force-pushed the exclude_and_strip branch from b53b1aa to 45e3cb5 Compare January 4, 2024 16:28

journalctl: add --exclude-identifier option

df40392

zhmylove force-pushed the exclude_and_strip branch from 45e3cb5 to df40392 Compare January 4, 2024 16:32

yuwata approved these changes Jan 4, 2024

View reviewed changes

yuwata added good-to-merge/waiting-for-ci 👍 PR is good to merge, but CI hasn't passed at time of review. Please merge if you see CI has passed and removed reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels Jan 4, 2024

bluca merged commit 25aa35d into systemd:main Jan 4, 2024

jackpot51 pushed a commit to pop-os/systemd that referenced this pull request May 7, 2024

autopkgtest: add bsdutils to upstream suite

a8fa075

The logger utility is now used in TEST-04-JOURNAL. See systemd/systemd#23086

Uh oh!

Conversation

zhmylove commented Apr 15, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Exclude syslog identifiers

Uh oh!

Uh oh!

bluca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhmylove commented Apr 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bluca commented Apr 15, 2022

Uh oh!

zhmylove commented Apr 15, 2022

Uh oh!

bluca commented Apr 15, 2022

Uh oh!

zhmylove commented Apr 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bluca commented Apr 16, 2022

Uh oh!

zhmylove commented Apr 16, 2022

Uh oh!

bluca commented Apr 16, 2022

Uh oh!

zhmylove commented Apr 16, 2022

Uh oh!

bluca commented Apr 16, 2022

Uh oh!

bluca commented Apr 16, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhmylove commented Jan 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuwata left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yuwata left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

10 participants

zhmylove commented Apr 15, 2022 •

edited by github-actions bot

Loading

zhmylove commented Apr 15, 2022 •

edited

Loading

zhmylove commented Apr 16, 2022 •

edited

Loading

zhmylove commented Jan 4, 2024 •

edited

Loading

yuwata left a comment •

edited

Loading