Describe the problem

See #72083 (comment) or any other issue linked to this thread, and roachdash

To Reproduce

Still reproducing, but we think it is related to restarting the roachprod vm and then trying to start CRDB again. Perhaps upon restarting, the /mnt/data1/cockroach permissions are not what they were on the initial boot.

@AlexTalks is taking a look.

Impact

Causing families of roachtests to fail predictably, see linked issues.