This repository was archived by the owner on Oct 16, 2020. It is now read-only.
This repository was archived by the owner on Oct 16, 2020. It is now read-only.
systemd-journald error causes 50+ load (normally 10) and stops logging. #322
Description
Problems:
1 - Journal stops recording messages
2 - Load on the node skyrockets
Recurrence:
Multiple times a day
wp-cos-003 ~ # journalctl -f
-- Logs begin at Sat 2015-04-04 16:40:41 UTC. --
Apr 08 02:25:42 wp-cos-003 docker[5576]: [2015-04-08 02:25:42,591: ERROR/Worker-6418] network_error: Cannot connect to host
Apr 08 02:25:42 wp-cos-003 docker[5576]: Traceback (most recent call last):
Apr 08 02:25:42 wp-cos-003 docker[5576]: File "/opt/crick/releases/0054c10264503446aac25ff4ee59b1f137d8c175/crick/tasks/queue.py", line 114, in _do_crawl
Apr 08 02:25:42 wp-cos-003 docker[5576]: ua_type=ua_type)
Apr 08 02:25:42 wp-cos-003 docker[5576]: File "/opt/crick/releases/0054c10264503446aac25ff4ee59b1f137d8c175/crick/api/controllers/query.py", line 291, in query_all
Apr 08 02:25:42 wp-cos-003 docker[5576]: crawled_page = crawl_url(url, ua_type)
Apr 08 02:25:42 wp-cos-003 docker[5576]: File "/opt/crick/releases/0054c10264503446aac25ff4ee59b1f137d8c175/crick/api/controllers/query.py", line 168, in crawl_url
Apr 08 02:25:42 wp-cos-003 docker[5576]: raise CrawlerSiteAuthorError(message)
Apr 08 02:25:42 wp-cos-003 docker[5576]: CrawlerSiteAuthorError: network_error: Cannot connect to host
Apr 08 02:25:42 wp-cos-003 docker[5576]: [2015-04-08 02:25:42,593: INFO/Worker-6418] Initializing redis connection pool on TCP:172.21.129.109:6391 [db:0]
Console and dmesg get spammed with failed to write entry messages:
wp-cos-003 ~ # dmesg
[43339.209945] systemd-journald[673]: Failed to write entry (18 items, 818 bytes), ignoring: Cannot assign requested address
[43339.221518] systemd-journald[673]: Failed to write entry (18 items, 862 bytes), ignoring: Cannot assign requested address
[43339.233061] systemd-journald[673]: Failed to write entry (18 items, 819 bytes), ignoring: Cannot assign requested address
[43339.526360] systemd-journald[673]: Failed to write entry (18 items, 858 bytes), ignoring: Cannot assign requested address
[43339.541282] systemd-journald[673]: Failed to write entry (18 items, 882 bytes), ignoring: Cannot assign requested address
.......
2500 more lines like above
.......
[43524.022977] systemd-journald[673]: Failed to write entry (18 items, 773 bytes), ignoring: Cannot assign requested address
[43524.034500] systemd-journald[673]: Failed to write entry (18 items, 669 bytes), ignoring: Cannot assign requested address
[43524.045984] systemd-journald[673]: Failed to write entry (18 items, 690 bytes), ignoring: Cannot assign requested address
[43524.057504] systemd-journald[673]: Failed to write entry (18 items, 644 bytes), ignoring: Cannot assign requested address
[43524.069037] systemd-journald[673]: Failed to write entry (18 items, 722 bytes), ignoring: Cannot assign requested address
[43524.080546] systemd-journald[673]: Failed to write entry (18 items, 626 bytes), ignoring: Cannot assign requested address
[43524.092081] systemd-journald[673]: Failed to write entry (18 items, 732 bytes), ignoring: Cannot assign requested address
[43524.103649] systemd-journald[673]: Failed to write entry (18 items, 648 bytes), ignoring: Cannot assign requested address
[43524.115194] systemd-journald[673]: Failed to write entry (18 items, 732 bytes), ignoring: Cannot assign requested address
[43524.132051] systemd-journald[673]: Failed to write entry (18 items, 647 bytes), ignoring: Cannot assign requested address
[43524.143572] systemd-journald[673]: Failed to write entry (18 items, 671 bytes), ignoring: Cannot assign requested address
[43524.155071] systemd-journald[673]: Failed to write entry (18 items, 768 bytes), ignoring: Cannot assign requested address
[43524.166565] systemd-journald[673]: Failed to write entry (18 items, 760 bytes), ignoring: Cannot assign requested address
[43524.178142] systemd-journald[673]: Failed to write entry (18 items, 898 bytes), ignoring: Cannot assign requested address
[43524.592761] systemd-journald[673]: Failed to write entry (18 items, 899 bytes), ignoring: Cannot assign requested address
[43524.936610] systemd-journald[673]: Failed to write entry (18 items, 898 bytes), ignoring: Cannot assign requested address
[43525.055179] systemd-journald[673]: Failed to write entry (18 items, 770 bytes), ignoring: Cannot assign requested address
Disk space / brts not an issue:
wp-cos-003 ~ # journalctl --disk-usage
Archived and active journals take up 3.9G on disk.
wp-cos-003 ~ # df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 24G 0 24G 0% /dev
tmpfs 24G 0 24G 0% /dev/shm
tmpfs 24G 3.0M 24G 1% /run
tmpfs 24G 0 24G 0% /sys/fs/cgroup
/dev/sda9 72G 11G 58G 16% /
/dev/sda4 985M 334M 601M 36% /usr
tmpfs 24G 0 24G 0% /media
tmpfs 24G 0 24G 0% /tmp
/dev/sda6 108M 68K 99M 1% /usr/share/oem
/dev/sr0 374K 374K 0 100% /media/configdrive
/dev/sdb1 630G 3.4G 627G 1% /var/lib/docker
wp-cos-003 ~ # df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 6175686 392 6175294 1% /dev
tmpfs 6187028 1 6187027 1% /dev/shm
tmpfs 6187028 1264 6185764 1% /run
tmpfs 6187028 14 6187014 1% /sys/fs/cgroup
/dev/sda9 4739808 106178 4633630 3% /
/dev/sda4 260096 5283 254813 3% /usr
tmpfs 6187028 2 6187026 1% /media
tmpfs 6187028 8 6187020 1% /tmp
/dev/sda6 32768 15 32753 1% /usr/share/oem
/dev/sr0 0 0 0 - /media/configdrive
/dev/sdb1 41943040 511906 41431134 2% /var/lib/docker
Verify Command:
wp-cos-003 ~ # journalctl --verify
e95fa0: (null) decompression failed: Bad message
e95fa0: invalid object contents: Bad message
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f67023-000513030a406af4.journal:e95fa0 (of 41943040 bytes, 36%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f67023-000513030a406af4.journal (Bad message)
76c71a0: invalid object
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000fa13b5-0005131340ec7fe6.journal:76c71a0 (of 125829120 bytes, 98%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000fa13b5-0005131340ec7fe6.journal (Cannot assign requested address)
7ffb0d0: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-00000000010b2601-00051328456c9353.journal
7f74250: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000cfdb1a-000512e9fb70cd34.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000001152983-0005132be772f0bd.journal
7fffe28: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000e47076-000512f89882f962.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-000000000101bfb3-00051325200c41da.journal
7ffff00: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000eea99e-000512fcab7226d9.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-00000000010da12d-000513292476cca5.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000e4986f-000512f8a8c0c80e.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000fc78fb-00051323041f7b3e.journal
000000: missing entry array
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000.journal:000000 (of 8388608 bytes, 0%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000.journal (Bad message)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000001064115-0005132694e3274c.journal
7fffe68: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000e996cc-000512faa86de579.journal
4b54d88: invalid object
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system.journal:4b54d88 (of 83886080 bytes, 94%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system.journal (Cannot assign requested address)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000ce0af6-000512e94e5ba149.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000d26628-000512ec47d9fb3e.journal
7fffe48: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000ec1c69-000512fbaa1cd509.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000fef00b-000513242cfd2f60.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f7787a-000513128d10345c.journal
32a8e10: (null) decompression failed: Bad message
32a8e10: invalid object contents: Bad message
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000001016081-0005132507b717ac.journal:32a8e10 (of 134217728 bytes, 39%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000001016081-0005132507b717ac.journal (Bad message)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000d5be82-000512f1655f644a.journal
7fffe98: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f3c7ea-000512fefa69a348.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000f495cf-000513003bf2ad3f.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000001105df2-0005132a1c5de7ae.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-000000000103d8c7-00051325c3a25c3c.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000e1da9d-000512f78adccbc3.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000df5f2c-000512f68bde359e.journal
7fffeb0: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000001101da2-0005132a05f262e5.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000dcb237-000512f577180322.journal
7fffe38: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000d50f10-000512f057c4def2.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000df3cca-000512f67db8b5ab.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-000000000108ce8e-000513276b912538.journal
7ff9b98: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-000000000115102f-0005132bddf50f07.journal
7fffe00: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000e1d75f-000512f78a54363c.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-000000000112989c-0005132af2fedb9c.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000cd4b83-000512e8fd58688c.journal
7ffff48: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000fc76f1-0005132303e3acf7.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000d35e82-000512edcf1ddc8d.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-00000000010da078-000513292467e738.journal
7fffe18: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000d7ad0e-000512f3f3dd6678.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000f6c9d6-0005130378088072.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000e708d1-000512f9a4627649.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000fb8b05-0005131401a55f04.journal
7fffec0: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000dca261-000512f57166fb86.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000cc9aac-000512e8b9c13116.journal
7fffea0: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000e7065d-000512f9a346ea0c.journal
7fffe40: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-00000000011296e0-0005132af27d1b3c.journal
7fffd78: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f13907-000512fdab311d0b.journal
65b28b0: (null) decompression failed: Bad message
65b28b0: invalid object contents: Bad message
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000da08b6-000512f46a364904.journal:65b28b0 (of 134217728 bytes, 79%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000da08b6-000512f46a364904.journal (Bad message)
7fffed8: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-000000000108b1a3-0005132760d916bb.journal
7fffe78: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000feec4c-000513242b875c8d.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000ed2d75-000512fc1b9ee023.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f71e50-00051312660d5cdb.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000da0d0a-000512f46b169af6.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000d8836b-000512f419dd34c7.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-00000000010b809f-000513286600498d.journal
7fffe68: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f77c5d-000513128dcdb093.journal
wp-cos-003 ~ # lsof -p $(pidof /usr/lib/systemd/systemd-journald)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-j 673 root cwd DIR 8,9 4096 2 /
systemd-j 673 root rtd DIR 8,9 4096 2 /
systemd-j 673 root txt REG 8,4 264720 34686 /usr/lib64/systemd/systemd-journald
systemd-j 673 root mem REG 8,9 83886080 6701 /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system.journal
systemd-j 673 root mem REG 8,9 8388608 6712 /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000.journal
systemd-j 673 root mem REG 8,4 18160 34196 /usr/lib64/libgpg-error.so.0.10.0
systemd-j 673 root mem REG 8,4 1787096 33884 /usr/lib64/libc-2.19.so
systemd-j 673 root mem REG 8,4 109288 33865 /usr/lib64/libpthread-2.19.so
systemd-j 673 root mem REG 8,4 31048 33873 /usr/lib64/librt-2.19.so
systemd-j 673 root mem REG 8,4 577128 34321 /usr/lib64/libgcrypt.so.11.8.2
systemd-j 673 root mem REG 8,4 153400 34193 /usr/lib64/liblzma.so.5.0.8
systemd-j 673 root mem REG 8,4 144720 33863 /usr/lib64/ld-2.19.so
systemd-j 673 root mem REG 0,18 8 12333 /run/systemd/journal/kernel-seqnum
systemd-j 673 root 0r CHR 1,3 0t0 1028 /dev/null
systemd-j 673 root 1w CHR 1,3 0t0 1028 /dev/null
systemd-j 673 root 2w CHR 1,3 0t0 1028 /dev/null
systemd-j 673 root 3u unix 0xffff880858da1180 0t0 24600 /run/systemd/journal/dev-log
systemd-j 673 root 4u unix 0xffff880858da0700 0t0 24592 /run/systemd/journal/stdout
systemd-j 673 root 5u unix 0xffff880858da0a80 0t0 24595 /run/systemd/journal/socket
systemd-j 673 root 6w CHR 1,11 0t0 1034 /dev/kmsg
systemd-j 673 root 7u 0000 0,10 0 8829 anon_inode
systemd-j 673 root 8u 0000 0,10 0 8829 anon_inode
systemd-j 673 root 9u CHR 1,11 0t0 1034 /dev/kmsg
systemd-j 673 root 10r REG 0,4 0 12334 /proc/sys/kernel/hostname
systemd-j 673 root 11u 0000 0,10 0 8829 anon_inode
systemd-j 673 root 12u 0000 0,10 0 8829 anon_inode
systemd-j 673 root 13u unix 0xffff88072dd4d400 0t0 5747188 /run/systemd/journal/stdout
systemd-j 673 root 14u unix 0xffff880858c27700 0t0 665 /run/systemd/journal/stdout
systemd-j 673 root 15u unix 0xffff880c687f5780 0t0 14527 /run/systemd/journal/stdout
systemd-j 673 root 16u unix 0xffff880c68a3e200 0t0 18145 /run/systemd/journal/stdout
systemd-j 673 root 17u unix 0xffff880c687b3100 0t0 14532 /run/systemd/journal/stdout
systemd-j 673 root 18u unix 0xffff880c687b6200 0t0 13494 /run/systemd/journal/stdout
systemd-j 673 root 19u unix 0xffff8806b52f3f00 0t0 22562921 /run/systemd/journal/stdout
systemd-j 673 root 20u unix 0xffff880c687b0380 0t0 13602 /run/systemd/journal/stdout
systemd-j 673 root 21u unix 0xffff880c687b4600 0t0 13509 /run/systemd/journal/stdout
systemd-j 673 root 22u unix 0xffff88084f5b3100 0t0 22251114 /run/systemd/journal/stdout
systemd-j 673 root 23u unix 0xffff880c687b3b80 0t0 13511 /run/systemd/journal/stdout
systemd-j 673 root 24u unix 0xffff880c686c6580 0t0 24991 /run/systemd/journal/stdout
systemd-j 673 root 25u unix 0xffff88084f5b2a00 0t0 22625095 /run/systemd/journal/stdout
systemd-j 673 root 26u unix 0xffff880c686c6c80 0t0 21731 /run/systemd/journal/stdout
systemd-j 673 root 27u unix 0xffff8805147c9180 0t0 22588496 /run/systemd/journal/stdout
systemd-j 673 root 28u unix 0xffff88085059a680 0t0 22280613 /run/systemd/journal/stdout
systemd-j 673 root 29u unix 0xffff8807162e5080 0t0 22599650 /run/systemd/journal/stdout
systemd-j 673 root 30u unix 0xffff88041e6ae900 0t0 22622230 /run/systemd/journal/stdout
systemd-j 673 root 31u unix 0xffff88057c5f5400 0t0 22604510 /run/systemd/journal/stdout
systemd-j 673 root 32u unix 0xffff880856b56200 0t0 22904697 /run/systemd/journal/stdout
systemd-j 673 root 33u unix 0xffff880001e0d400 0t0 22904698 /run/systemd/journal/stdout
systemd-j 673 root 34u unix 0xffff8806ca0f8700 0t0 22904699 /run/systemd/journal/stdout
systemd-j 673 root 35u unix 0xffff880858e35b00 0t0 22686224 /run/systemd/journal/stdout
systemd-j 673 root 36u unix 0xffff880858136200 0t0 22686225 /run/systemd/journal/stdout
systemd-j 673 root 37u unix 0xffff88057c708380 0t0 22686226 /run/systemd/journal/stdout
systemd-j 673 root 38u unix 0xffff8806a69a8000 0t0 22692030 /run/systemd/journal/stdout
systemd-j 673 root 39u unix 0xffff88085059bf00 0t0 22708104 /run/systemd/journal/stdout
systemd-j 673 root 40u unix 0xffff88085059b480 0t0 22704104 /run/systemd/journal/stdout
systemd-j 673 root 41u unix 0xffff880945269500 0t0 22715904 /run/systemd/journal/stdout
systemd-j 673 root 42u unix 0xffff880557e0d080 0t0 22714438 /run/systemd/journal/stdout
systemd-j 673 root 43u unix 0xffff880557e09f80 0t0 22714439 /run/systemd/journal/stdout
systemd-j 673 root 44u unix 0xffff8807162e6c80 0t0 22742614 /run/systemd/journal/stdout
systemd-j 673 root 45u unix 0xffff880523246580 0t0 22776062 /run/systemd/journal/stdout
systemd-j 673 root 46u unix 0xffff880851e12d80 0t0 22776063 /run/systemd/journal/stdout
systemd-j 673 root 47u unix 0xffff880851fbb480 0t0 22776064 /run/systemd/journal/stdout
systemd-j 673 root 48u unix 0xffff880856b57380 0t0 22776107 /run/systemd/journal/stdout
systemd-j 673 root 49u unix 0xffff880523245780 0t0 22808741 /run/systemd/journal/stdout
systemd-j 673 root 50u unix 0xffff880c6804bb80 0t0 22776108 /run/systemd/journal/stdout
systemd-j 673 root 51u unix 0xffff880c66ea5400 0t0 22808742 /run/systemd/journal/stdout
systemd-j 673 root 52u unix 0xffff880c66ea0000 0t0 22808743 /run/systemd/journal/stdout
systemd-j 673 root 53u unix 0xffff8808574a6580 0t0 22812590 /run/systemd/journal/stdout
systemd-j 673 root 54u unix 0xffff8808574a4600 0t0 22812591 /run/systemd/journal/stdout
systemd-j 673 root 55u unix 0xffff880850b9ec80 0t0 22844553 /run/systemd/journal/stdout
systemd-j 673 root 56u unix 0xffff880851fb9880 0t0 22826885 /run/systemd/journal/stdout
systemd-j 673 root 57u unix 0xffff8808580ead80 0t0 22855102 /run/systemd/journal/stdout
systemd-j 673 root 58u unix 0xffff8808503de200 0t0 22861991 /run/systemd/journal/stdout
systemd-j 673 root 59u unix 0xffff880c66ea5780 0t0 22861992 /run/systemd/journal/stdout
systemd-j 673 root 60u unix 0xffff88084d9e9f80 0t0 22868703 /run/systemd/journal/stdout
systemd-j 673 root 61u unix 0xffff88057c517380 0t0 22864259 /run/systemd/journal/stdout
systemd-j 673 root 62u unix 0xffff880523241f80 0t0 22868704 /run/systemd/journal/stdout
systemd-j 673 root 63u unix 0xffff880523244980 0t0 22868705 /run/systemd/journal/stdout
systemd-j 673 root 64u unix 0xffff880858ea5080 0t0 25186561 /run/systemd/journal/stdout
systemd-j 673 root 65u unix 0xffff88084cd71f80 0t0 22931995 /run/systemd/journal/stdout
systemd-j 673 root 66u unix 0xffff88085059bb80 0t0 22931996 /run/systemd/journal/stdout
systemd-j 673 root 67u unix 0xffff88084c450000 0t0 22931997 /run/systemd/journal/stdout
systemd-j 673 root 68u unix 0xffff880001e0c280 0t0 22904700 /run/systemd/journal/stdout
systemd-j 673 root 69u unix 0xffff88057c70bb80 0t0 22281130 /run/systemd/journal/stdout
systemd-j 673 root 70u REG 8,9 83886080 6701 /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system.journal
systemd-j 673 root 71u unix 0xffff88085a6d2680 0t0 22284303 /run/systemd/journal/stdout
systemd-j 673 root 72u unix 0xffff880c68119500 0t0 22654482 /run/systemd/journal/stdout
systemd-j 673 root 73u unix 0xffff880c68258a80 0t0 25280427 /run/systemd/journal/stdout
systemd-j 673 root 74u unix 0xffff88057c5f3800 0t0 22307568 /run/systemd/journal/stdout
systemd-j 673 root 75u unix 0xffff88057c709880 0t0 22307569 /run/systemd/journal/stdout
systemd-j 673 root 76u REG 8,9 8388608 6712 /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000.journal
systemd-j 673 root 77u unix 0xffff880858ea2680 0t0 25204054 /run/systemd/journal/stdout
The only solution I have found to bring load back under control and restore logging (tbh the load issue is the main problem here) is forcing the process to restart by kill -SIGUSR2 $(pidof /usr/lib/systemd/systemd-journald)