Skip to content

Fixes for device activation while reloading and switching root#23215

Closed
mwilck wants to merge 2 commits intosystemd:mainfrom
mwilck:issue23208
Closed

Fixes for device activation while reloading and switching root#23215
mwilck wants to merge 2 commits intosystemd:mainfrom
mwilck:issue23208

Conversation

@mwilck
Copy link
Copy Markdown
Contributor

@mwilck mwilck commented Apr 28, 2022

This PR fixes two issues related to device activation.

  1. When systemd was being reloaded during boot (e.g. systemctl reload run by some service), devices state could switch from "plugged" to "dead" and back, causing spurious umounts (Devices transition plugged -> dead -> plugged is still not completely fixed #12953)
  2. When the state of a previously "plugged" device changed because of SYSTEMD_READY=0 after switching root (because of change udev rules files), systemd might prematurely activate the device even though wasn't ready to use. This happens in multipath setups and can lead to boot failure (Boot to emergency mode if devices are switched to SYSTEMD_READY=0 (multipath scenario) #23208).

This PR fixes both issues.

  1. In the "reload" case, the udev found flag shouldn't be reset. The device monitor socket is live during the reload operation, so if a previously found device is not seen any more, it will either be re-detected, or systemd will receive a REMOVE uevent.
  2. In the switch root case, we can be sure that systemd-udev-trigger.service will be re-run. The state of plugged devices should be reset to DEVICE_TENTATIVE to avoid premature activation. In the likely case that the device is still plugged, it will be re-detected by udev. This way changed udev properties can be picked up correctly by systemd.

More detailed rationale is found in the commit descriptions, comments, and in the two referenced issues.

Fixes #12953
Fixes #23208

While the manager is reloaded, the device_monitor socket is kept
open; outstanding uevents will be received sooner or later. Thus if a device
isn't found during udev enumeration / coldplug, we shouldn't blindly assume
it's gone. Possibly the reload operation took place before udev started, or had
the time to re-probe previously discovered devices (this matters mainly during
boot / coldplug). So if the device actually disappeared, leave it to the
outstanding uevent to make this happen in systemd.

Fixes systemd#12953.
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 29, 2022
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 29, 2022
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 29, 2022
When we switch root, it can happen (e.g. because of different udev
rules) that a previously plugged device will change to "dead"
(SYSTEMD_READY=0). The udev db is actually cleared and will be
built up from scratch. We shouldn't activate any mounts or swaps
bound to previously "plugged" devices before we actually receive
uevents for them indicating they're ready to use.

Hence, when we deserialize device units in the "switching root"
case (which we can detect from manager->honor_device_enumeration),
clear the DEVICE_FOUND_UDEV flag, and map DEVICE_PLUGGED to
DEVICE_TENTATIVE. Thus devices which are neither mounted nor used
for swap will startup as DEVICE_NOT_FOUND / DEVICE_DEAD, and mounted
devices as DEVICE_FOUND_MOUNT / DEVICE_TENTATIVE. This should play
well with the logic in device_found_changed().

It is expected that udev coldplug will happen some time after switching root,
and that the actual device state will be set by device_dispatch_io(),
which will also activate dependent mount units.

Fixes systemd#23208.
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 29, 2022
See the comments in the code.

This is based on the work by Martin Wilck.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 29, 2022
See the comments in the code.

This is based on the work by Martin Wilck.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 29, 2022
See the comments in the code.

This is based on the work by Martin Wilck.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.
@yuwata yuwata added the pid1 label Apr 29, 2022
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 30, 2022
See the comments in the code.

This is based on the work by Martin Wilck.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 30, 2022
See the comments in the code.

This is based on the work by Martin Wilck.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.
yuwata added a commit to yuwata/systemd that referenced this pull request Apr 30, 2022
See the comments in the code.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>
yuwata added a commit to yuwata/systemd that referenced this pull request May 1, 2022
The issue systemd#12953 is caused by the following:
On switching root,
- deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT,
- deserialized_state == DEVICE_PLUGGED,
- enumerated_found == DEVICE_FOUND_MOUNT,
On switching root, most devices are not found by the enumeration process.
Hence, the device state is set to plugged by device_coldplug(), and then
changed to the dead state in device_catchup(). So the corresponding
mount point is unmounted. Later when the device is processed by udevd, it
will be changed to plugged state again.

The issue systemd#23208 is caused by the fact that generated udev database in
initramfs and the main system are often different.

So, the two issues have the same root; we should not honor
DEVICE_FOUND_UDEV bit in the deserialized_found on switching root.

This partially reverts c6e892b.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>
yuwata added a commit to yuwata/systemd that referenced this pull request May 1, 2022
The issue systemd#12953 is caused by the following:
On switching root,
- deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT,
- deserialized_state == DEVICE_PLUGGED,
- enumerated_found == DEVICE_FOUND_MOUNT,
On switching root, most devices are not found by the enumeration process.
Hence, the device state is set to plugged by device_coldplug(), and then
changed to the dead state in device_catchup(). So the corresponding
mount point is unmounted. Later when the device is processed by udevd, it
will be changed to plugged state again.

The issue systemd#23208 is caused by the fact that generated udev database in
initramfs and the main system are often different.

So, the two issues have the same root; we should not honor
DEVICE_FOUND_UDEV bit in the deserialized_found on switching root.

This partially reverts c6e892b.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>
yuwata added a commit to yuwata/systemd that referenced this pull request May 1, 2022
The issue systemd#12953 is caused by the following:
On switching root,
- deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT,
- deserialized_state == DEVICE_PLUGGED,
- enumerated_found == DEVICE_FOUND_MOUNT,
On switching root, most devices are not found by the enumeration process.
Hence, the device state is set to plugged by device_coldplug(), and then
changed to the dead state in device_catchup(). So the corresponding
mount point is unmounted. Later when the device is processed by udevd, it
will be changed to plugged state again.

The issue systemd#23208 is caused by the fact that generated udev database in
initramfs and the main system are often different.

So, the two issues have the same root; we should not honor
DEVICE_FOUND_UDEV bit in the deserialized_found on switching root.

This partially reverts c6e892b.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>
yuwata added a commit to yuwata/systemd that referenced this pull request May 2, 2022
The issue systemd#12953 is caused by the following:
On switching root,
- deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT,
- deserialized_state == DEVICE_PLUGGED,
- enumerated_found == DEVICE_FOUND_MOUNT,
On switching root, most devices are not found by the enumeration process.
Hence, the device state is set to plugged by device_coldplug(), and then
changed to the dead state in device_catchup(). So the corresponding
mount point is unmounted. Later when the device is processed by udevd, it
will be changed to plugged state again.

The issue systemd#23208 is caused by the fact that generated udev database in
initramfs and the main system are often different.

So, the two issues have the same root; we should not honor
DEVICE_FOUND_UDEV bit in the deserialized_found on switching root.

This partially reverts c6e892b.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>
yuwata added a commit to yuwata/systemd that referenced this pull request May 11, 2022
The issue systemd#12953 is caused by the following:
On switching root,
- deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT,
- deserialized_state == DEVICE_PLUGGED,
- enumerated_found == DEVICE_FOUND_MOUNT,
On switching root, most devices are not found by the enumeration process.
Hence, the device state is set to plugged by device_coldplug(), and then
changed to the dead state in device_catchup(). So the corresponding
mount point is unmounted. Later when the device is processed by udevd, it
will be changed to plugged state again.

The issue systemd#23208 is caused by the fact that generated udev database in
initramfs and the main system are often different.

So, the two issues have the same root; we should not honor
DEVICE_FOUND_UDEV bit in the deserialized_found on switching root.

This partially reverts c6e892b.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>
@keszybz
Copy link
Copy Markdown
Member

keszybz commented May 12, 2022

This is superseded by #23218, right?

@mwilck
Copy link
Copy Markdown
Contributor Author

mwilck commented May 12, 2022

This is superseded by #23218, right?

yes. Closing this.

@mwilck mwilck closed this May 12, 2022
yuwata added a commit to yuwata/systemd that referenced this pull request May 12, 2022
The issue systemd#12953 is caused by the following:
On switching root,
- deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT,
- deserialized_state == DEVICE_PLUGGED,
- enumerated_found == DEVICE_FOUND_MOUNT,
On switching root, most devices are not found by the enumeration process.
Hence, the device state is set to plugged by device_coldplug(), and then
changed to the dead state in device_catchup(). So the corresponding
mount point is unmounted. Later when the device is processed by udevd, it
will be changed to plugged state again.

The issue systemd#23208 is caused by the fact that generated udev database in
initramfs and the main system are often different.

So, the two issues have the same root; we should not honor
DEVICE_FOUND_UDEV bit in the deserialized_found on switching root.

This partially reverts c6e892b.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>
bluca pushed a commit to bluca/systemd that referenced this pull request Jan 27, 2023
The issue systemd#12953 is caused by the following:
On switching root,
- deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT,
- deserialized_state == DEVICE_PLUGGED,
- enumerated_found == DEVICE_FOUND_MOUNT,
On switching root, most devices are not found by the enumeration process.
Hence, the device state is set to plugged by device_coldplug(), and then
changed to the dead state in device_catchup(). So the corresponding
mount point is unmounted. Later when the device is processed by udevd, it
will be changed to plugged state again.

The issue systemd#23208 is caused by the fact that generated udev database in
initramfs and the main system are often different.

So, the two issues have the same root; we should not honor
DEVICE_FOUND_UDEV bit in the deserialized_found on switching root.

This partially reverts c6e892b.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>
(cherry picked from commit 75d7b59)
Werkov pushed a commit to Werkov/systemd that referenced this pull request Nov 1, 2023
The issue systemd#12953 is caused by the following:
On switching root,
- deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT,
- deserialized_state == DEVICE_PLUGGED,
- enumerated_found == DEVICE_FOUND_MOUNT,
On switching root, most devices are not found by the enumeration process.
Hence, the device state is set to plugged by device_coldplug(), and then
changed to the dead state in device_catchup(). So the corresponding
mount point is unmounted. Later when the device is processed by udevd, it
will be changed to plugged state again.

The issue systemd#23208 is caused by the fact that generated udev database in
initramfs and the main system are often different.

So, the two issues have the same root; we should not honor
DEVICE_FOUND_UDEV bit in the deserialized_found on switching root.

This partially reverts c6e892b.

Fixes systemd#12953 and systemd#23208.
Replaces systemd#23215.

Co-authored-by: Martin Wilck <mwilck@suse.com>

(cherry picked from commit 75d7b59)

[mwilck: fixes bsc#1137373]
[mwilck: fixes bsc#1181658]
[mwilck: fixes bsc#1194708]
[mwilck: fixes bsc#1195157]
[mwilck: fixes bsc#1197570]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3 participants