Describe the bug

On a network-based file system like OpenAFS, hard-linking files in different directories would cause the "cross-device link" error (EXDEV).

When building a derivation inside the (unprivileged) local chroot store, Nix will first create an in-store directory with the .chroot suffix (chrootRootDir), and hard link files into this directory with the function linkOrCopy. linkOrCopy will only fallback to copying upon EMLIINK or EPERM, causing the build to fail.

The following output is got when trying to build derivations using the statically-linked Nix binary on CERN AFS on CERN LXPLUS.

[yuehshun@lxplus810 optimization]$ nix --extra-experimental-features "nix-command flakes" --ssl-cert-file ~/.local/share/nix/ca-bundle-nix.crt run -L --no-sandbox path:count-to-60.tmp
error: linking '/afs/cern.ch/user/y/yuehshun/.local/share/nix/root/nix/store/ylmzfscr0p5glfh4d4sh22hhl9lwi0jb-count-to-60.drv.chroot/nix/store/5yzw0vhkyszf2d179m0qfkgxmp5wjjx4-move-docs.sh' to '/afs/cern.ch/user/y/yuehshun/.local/share/nix/root/nix/store/5yzw0vhkyszf2d179m0qfkgxmp5wjjx4-move-docs.sh': Cross-device link

This issue may prevent HPC users from building the derivation on the server.

Steps To Reproduce

1. Act as an unprivileged user
2. cd onto an OpenAFS instance (Andrew File System)
3. Place the statically-linked Nix binary under ~/.local/bin
4. Place the ca-bundle.crt got from the package cacert $out/etc/ssl/certs/ca-bundle.crt as ~/.local/share/nix/ca-bundle-nix.crt
5. Create a directory named hello-flake
6. Create hello-flake/flake.nix with the following content

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs";
  inputs.flake-parts.url = "github:hercules-ci/flake-parts";
  inputs.systems.url = "github:nix-systems/default";
  outputs = inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      perSystem = { config, lib, pkgs, ... }:
      {
        packages.say-hello = pkgs.writeShellScriptBin "say-hello" { } ''
          echo "Hello Nix!"
          date
        '';
        packages.default = config.packages.say-hello;
      };
      systems = import inputs.systems;
    };
}

6. Run

nix --extra-experimental-features "nix-command flakes" --ssl-cert-file ~/.local/share/nix/ca-bundle-nix.crt run -L path:hello-flake

7. See error.

Expected behavior

Unprivileged users be able to build derivations on network-based file system where hard link across directories are not supported.

The quickest fix would be letting linkOrCopy fall back to copy upon EXDEV, as it already does upon EPERM. It would be more ideal if those dependency could be bind-mount instead of copy into chrootRootDir.

nix-env --version output
nix (Nix) 2.15.0

Additional context

Add any other context about the problem here.

Priorities

Add 👍 to issues you find important.