Describe the bug

hydra-queue-runner can begin to to fail with:

--- SysError --- hydra-queue-runner
creating directory '/var/lib/hydra/build-logs': Permission denied

… when the system is activated. Specifically, this bug occurs when deploying/upgrading a system in such a way that the hydra-init script is not re-run (e.g. no changes to the NixOS hydra module that would trigger a reload).

To Reproduce

Steps to reproduce the behavior:

1. Enable the hydra service on a NixOS system
2. Run /run/current-system/activate as root
3. hydra-queue-runner will begin failing with a permissions error on the next build

Root cause

I've already root-caused the origin of the problem, which was first introduced in 8833983. That commit changed the update-users-groups.pl script to always chmod 700 ~ for each system user.

The problem arises because the hydra user's home directory is set to /var/lib/hydra, so when the system is activated the permissions on /var/lib/hydra are set to 700, which causes hydra-queue-runner to fail (since the permissions are supposed to be 750 for proper operation).

Workarounds

If you're reading this issue and you're affected by this problem, a short-term solution is to restart the hydra-init service:

sudo systemctl restart hydra-init

This will fix the system until the next time you upgrade the system without upgrading hydra.

Possible long-term solution

I think one way to fix this issue is to not specify createHome for the hydra user. The reason why is that the hydra-init script already creates the /var/lib/hydra directory, so it shouldn't be necessary for createHome to be specified.