Sandbox doesn't work on macOS Big Sur when the sandbox directory is in a firm linked directory #13766
Description
macOS Catalina (10.*) introduced a security feature where the root filesystem is mounted read-only, and a mechanism for creating directories or symlinks at boot time called firmlinks. Since the root filesystem is read-only, a new volume mounted at /System/Volumes/Data is overlaid on top of the root filesystem and contains the mutable portions of the filesystem (more info here).
When building with Bazel on a location that is firm linked from the root, and moving the sandbox to that same location by using the --experimental_sandbox_base flag:
In macOS Catalina (10.15.7 to be precise), it works as expected.
$ cat /etc/synthetic.conf
export /System/Volumes/Data/export
$ ls -l /export
lrwxr-xr-x 1 root wheel 27 May 30 22:42 /export -> /System/Volumes/Data/export
$ cd /export/home/tester
$ git clone https://github.com/bazelbuild/bazel.git
Cloning into 'bazel'...
remote: Enumerating objects: 567875, done.
remote: Counting objects: 100% (1761/1761), done.
remote: Compressing objects: 100% (924/924), done.
remote: Total 567875 (delta 783), reused 1374 (delta 586), pack-reused 566114
Receiving objects: 100% (567875/567875), 757.61 MiB | 27.05 MiB/s, done.
Resolving deltas: 100% (354168/354168), done.
Updating files: 100% (9123/9123), done.
$ cd bazel
$ bazel build --experimental_sandbox_base=$(pwd)/sandbox-dir //src:bazel
INFO: Elapsed time: 92.474s, Critical Path: 29.31s
INFO: 2915 processes: 2034 remote cache hit, 167 internal, 533 darwin-sandbox, 1 local, 180 worker.
INFO: Build completed successfully, 2915 total actions
However, that exact same sequence of operations on macOS Big Sur (11.4), produces the following error:
$ bazel build --experimental_sandbox_base=$(pwd)/sandbox-dir //src:bazel
<output elided>
ERROR: /System/Volumes/Data/export/home/tester/bazel/third_party/BUILD:467:20: Extracting interface //third_party:flogger_checked_in failed: (Segmentation fault): ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/flogger/flogger-0.5.1.jar ... (remaining 3 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/flogger/flogger-0.5.1.jar ... (remaining 3 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
open(): Operation not permitted
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /System/Volumes/Data/export/home/tester/bazel/src/main/java/com/google/devtools/build/lib/analysis/BUILD:2085:13 Building src/main/java/com/google/devtools/build/lib/analysis/libextra/extra_action_info_file_write_action.jar (1 source file) and running annotation processors (AutoCodecProcessor) failed: (Segmentation fault): ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/tomcat_annotations_api/tomcat-annotations-api-8.0.5.jar ... (remaining 3 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/tomcat_annotations_api/tomcat-annotations-api-8.0.5.jar ... (remaining 3 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
INFO: Elapsed time: 14.914s, Critical Path: 0.17s
INFO: 23 processes: 1 remote cache hit, 22 internal.
FAILED: Build did NOT complete successfully
I have traced the problem to a difference on how sandbox-exec works in Big Sur when firm links are involved.
If I create the following sandbox.sb file (modified from the actual sandbox.sb that Bazel creates):
(version 1)
(debug deny)
(allow default)
(deny file-write*)
(allow file-write*
(subpath "/dev")
(subpath "/System/Volumes/Data/export/home/tester/Library/Logs")
(subpath "/private/var/tmp")
(subpath "/private/var/folders/p1/5rdx8yks6wl56p49qhzlnbv40000gq/C")
(subpath "/private/var/folders/p1/5rdx8yks6wl56p49qhzlnbv40000gq/T")
(subpath "/System/Volumes/Data/export/home/tester/sb/sandbox-dir")
(subpath "/System/Volumes/Data/export/home/tester/TEMP")
(subpath "/private/tmp")
(subpath "/System/Volumes/Data/export/home/tester/Library/Developer")
)
And put it in an empty directory in /export/home/tester/sb, on macOS Catalina:
$ mkdir sandbox-dir
$ /usr/bin/sandbox-exec -f sandbox.sb /bin/bash -c "touch sandbox-dir/foobar"
$ echo $?
0
$ ls -l sandbox-dir/
total 0
-rw-r--r-- 1 tester staff 0 Jul 28 10:08 foobar
Everything works as expected. However on macOS Big Sur:
$ mkdir sandbox-dir
$ /usr/bin/sandbox-exec -f sandbox.sb /bin/bash -c "touch sandbox-dir/foobar"
touch: sandbox-dir/foobar: Operation not permitted
$ ls -l sandbox-dir/
However, if I change the line:
(subpath "/System/Volumes/Data/export/home/tester/sb/sandbox-dir")
to remove the /System/Volumes/Data prefix (since I'm using that firm link through the symlink), I get the expected results in macOS Big Sur:
Now, this could be a bug in how macOS handles firm links but Bazel could also avoid resolving symlinks (or firm links) on macOS to work around this. I will file a bug with Apple but the man page of sandbox-exec says the command is deprecated so I don't have high hopes there.