systemd version the issue has been seen with

v239

Used distribution

embedded system (but also reproduced with Debian and systemd v239)

Expected behavior you didn't see

serialized/deserialized device jobs don't hang forever after "daemon-reload"

Unexpected behavior you saw

serialized/deserialized device jobs hang forever after "daemon-reload"

Steps to reproduce the problem

To be executed in two different shells, a couple of iterations are enough. With proper timing this issue can be also reproduced with just one "udevadm trigger" & "systemctl daemon-reload" call in parallel if the latter is delayed a bit.

$ while true; do systemctl daemon-reload; done
$ while true; do udevadm trigger; done

If running device jobs get intercepted by "systemctl daemon-reload" (and in turn get serialized/deserialized), these will hang forever after "daemon-reload" returned. These device jobs hang in state "nop waiting" forever as can be seen afterwards with "systemctl list-jobs".

I've bisected this behavior down to the following commit:

f332611abee97247e40311d99492d53697152a1a is the first bad commit
commit f332611abee97247e40311d99492d53697152a1a
Author: Boucman <jeremy.rosen@enst-bretagne.fr>
Date: Fri Nov 10 17:00:32 2017 +0100

device : reload when udev generates a "changed" event (#6850)

This commit isn't the root cause of the problem but the one which caused to reproduce the issue easily with the above mentioned procedure. In general it seems that either all device jobs or "nop waiting" jobs which get serialized/deserialized hang forever after "daemon-reload" returns.

This issue can easily be reproduced, are further debug logs needed?