Description of the problem / feature request:

When using the new --incompatible_remote_build_event_upload_respect_no_cache flag, along with a combined remote+disk cache, blobs that are only for BES upload are uploaded to the disk cache (in addition to the remote cache). This has a couple negative side effects including:

• Using disk space for blobs that will never be referenced locally
• Performing expensive FindMissingBlobs calls on the disk cache
• Causing --bes_upload_mode=fully_async to block for disk IO

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Perform a build with the following flags, two times:

build --bes_upload_mode=fully_async
build --remote_cache=grpcs://remote.buildbuddy.io
build --bes_results_url=https://app.buildbuddy.io/invocation/
build --bes_backend=grpcs://remote.buildbuddy.io
build --disk_cache=/tmp/bazel_disk_cache
build --incompatible_remote_build_event_upload_respect_no_cache
build --noremote_upload_local_results

If the build has lots of things that will be BES-only uploads (one of my examples was 20k+ blobs), the second build will stall at the end as it performs disk cache related operations.

What operating system are you running Bazel on?

macOS 12.0.1

What's the output of bazel info release?

release 5.0.0rc3

Have you found anything relevant by searching the web?

#14338