Skip to content

dracut/99setup-root: change owner of /run/log/journal#59

Merged
tormath1 merged 2 commits intoflatcar-masterfrom
tormath1/journald
Mar 30, 2023
Merged

dracut/99setup-root: change owner of /run/log/journal#59
tormath1 merged 2 commits intoflatcar-masterfrom
tormath1/journald

Conversation

@tormath1
Copy link
Copy Markdown
Contributor

Give the correct ownernship otherwise the tmpfiles configuration will not apply because of mismatching machine-id - which makes "journalctl --user" to fail


# Give the correct ownernship otherwise the tmpfiles configuration will not
# apply because of mismatching machine-id - which makes "journalctl --user" to fail.
chown --recursive root:systemd-journal /run/log/journal/
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, this reminds me of flatcar/Flatcar#799
I think the problem could be that systemd-tmpfiles doesn't find the systemd-journal group because it's not part of /sysroot/etc/group but only loaded via the nss-altfiles plugin from our database in baselayout.
I think as long as we don't change systemd-tmpfiles to somehow resolve the nss stuff in the given chroot, we should all systemd-journal as special group to copy like docker: https://github.com/flatcar/baselayout/blob/flatcar-master/scripts/flatcar-tmpfiles#L13
This will also be a workaround for the linked issue (at least for the journal but not for kvm - maybe add that, too? At some point we should fix the tools, though).

Copy link
Copy Markdown
Member

@pothos pothos Mar 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the assumption that this existing tmpfiles rule here would set the group:

/usr/lib/tmpfiles.d/systemd.conf:z /run/log/journal 2755 root systemd-journal - -
/usr/lib/tmpfiles.d/systemd.conf:Z /run/log/journal/%m ~2750 root systemd-journal - -

Copy link
Copy Markdown
Contributor Author

@tormath1 tormath1 Mar 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I checked this initially and the issue is not to change the owner in /sysroot/run/log/journal but in /run/log/journal (the one in the initramfs) so I tried with systemd-tmpfiles --create systemd.conf but in the end, I thought it would be overkill so just changed the ownership of the /run/log/journal.

Copy link
Copy Markdown
Member

@pothos pothos Mar 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok, right, /sysroot/run isn't mounted and this setup would take place later and not in the call we do here.
If this is about /run/log/journal of the initrd I think it's at the wrong place. Edit: Maybe 99journald-conf?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, found this here: b0b7416

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried it out manually with rd.shell rd.break=cmdline (=pre-mount or =pre-pivot would be too late) on a qemu arm64 VM.
After the VM booted I see this:

core@localhost ~ $ ls -alh /run/log/
total 0
drwxr-xr-x.  3 root root             60 Mar 28 03:10 .
drwxr-xr-x. 30 root root            780 Mar 28 03:16 ..
drwxr-sr-x.  4 root systemd-journal  80 Mar 28 03:15 journal
core@localhost ~ $ ls -alh /run/log/journal/
total 0
drwxr-sr-x. 4 root systemd-journal 80 Mar 28 03:15 .
drwxr-xr-x. 3 root root            60 Mar 28 03:10 ..
drwxr-s---. 2 root systemd-journal 60 Mar 28 03:10 931d6b0b22a24775bbd156cfaf0e98f4
drwxr-s---. 2 root systemd-journal 60 Mar 28 03:15 bd9ce2b134744c6db1aca45238fd9848
core@localhost ~ $ ls -alh /run/log/journal/*
/run/log/journal/931d6b0b22a24775bbd156cfaf0e98f4:
total 2.5M
drwxr-s---. 2 root systemd-journal   60 Mar 28 03:10 .
drwxr-sr-x. 4 root systemd-journal   80 Mar 28 03:15 ..
-rw-r-----. 1 root systemd-journal 2.5M Mar 28 03:14 system.journal

/run/log/journal/bd9ce2b134744c6db1aca45238fd9848:
total 2.5M
drwxr-s---. 2 root systemd-journal   60 Mar 28 03:15 .
drwxr-sr-x. 4 root systemd-journal   80 Mar 28 03:15 ..
-rw-r-----. 1 root systemd-journal 2.5M Mar 28 03:16 system.journal
core@localhost ~ $ ls -alh /run/log/journal/*/*
-rw-r-----. 1 root systemd-journal 2.5M Mar 28 03:14 /run/log/journal/931d6b0b22a24775bbd156cfaf0e98f4/system.journal
-rw-r-----. 1 root systemd-journal 2.5M Mar 28 03:16 /run/log/journal/bd9ce2b134744c6db1aca45238fd9848/system.journal

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I have no answer for why this happens only on arm64 and why the upstream file wants local-fs - probably more for the real system and not really needed in the initrd, but no idea how to encode this conditionally)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is done. AMD64 is failing too for the very same reason. Ignition's run seems to be the point of failure (as it pass the update test)

Mar 27 19:56:35.796020 systemd[1]: ignition-fetch.service: Found ordering cycle on systemd-resolved.service/start
Mar 27 19:56:35.796028 systemd[1]: ignition-fetch.service: Found dependency on systemd-tmpfiles-setup.service/start
Mar 27 19:56:35.796036 systemd[1]: ignition-fetch.service: Found dependency on local-fs.target/start
Mar 27 19:56:35.796043 systemd[1]: ignition-fetch.service: Found dependency on local-fs-pre.target/start
Mar 27 19:56:35.796051 systemd[1]: ignition-fetch.service: Found dependency on ignition-disks.service/start
Mar 27 19:56:35.796058 systemd[1]: ignition-fetch.service: Found dependency on ignition-fetch.service/start

It looks like we differ from coreos/ignition on this:

# Flatcar:
Wants=systemd-networkd.service
After=systemd-networkd.service
Wants=systemd-resolved.service
After=systemd-resolved.service
Requires=local-fs-pre.target
Before=local-fs-pre.target

(https://github.com/coreos/ignition/blob/main/dracut/30ignition/ignition-fetch.service &

# If we run, we definitely need network, so make sure we run after.
After=network.target
# Flatcar:
Wants=systemd-networkd.service
After=systemd-networkd.service
Wants=systemd-resolved.service
After=systemd-resolved.service
Requires=local-fs-pre.target
Before=local-fs-pre.target
Requires=ignition-setup.service
After=ignition-setup.service
)

Do you remember why we have this dependency:

 Requires=local-fs-pre.target 
 Before=local-fs-pre.target 

?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As said, I think we can tweak the service to run a bit earlier, too. I've moved the comment into a suggestion.

@tormath1 tormath1 force-pushed the tormath1/journald branch from fcd7e86 to 9aceb99 Compare March 27, 2023 13:50
@tormath1 tormath1 force-pushed the tormath1/journald branch from 8d11a68 to d45435c Compare March 29, 2023 16:13
tormath1 and others added 2 commits March 30, 2023 14:38
otherwise it leads to a cyclic dependency with ignition-fetch.service

```
Found ordering cycle on systemd-resolved.service/start
Found dependency on systemd-tmpfiles-setup.service/start
Found dependency on local-fs.target/start
Found dependency on local-fs-pre.target/start
Found dependency on ignition-disks.service/start
Found dependency on ignition-fetch.service/start
```

Co-authored-by: Kai Lüke <pothos@users.noreply.github.com>
Signed-off-by: Mathieu Tortuyaux <mtortuyaux@microsoft.com>
@tormath1 tormath1 force-pushed the tormath1/journald branch from d45435c to 66b6dda Compare March 30, 2023 12:41
@tormath1 tormath1 merged commit dcd4b1e into flatcar-master Mar 30, 2023
@tormath1 tormath1 deleted the tormath1/journald branch March 30, 2023 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants