At the time of the first compaction, rocksdb detects a bad checksum:
E180530 21:41:29.961699 230 storage/replica.go:4827 [n1,s1,r33/1:/Table/53/1/-8{15009…-09684…}] unable to update the state machine: could not commit batch: Corruption: block checksum mismatch: expected 4061029206, got 769420182 in /mnt/data1/cockroach/000007.sst offset 710670 size 14828
This is happening on the release binaries. Oddly enough, it is not happening on the dev binaries build on my local machine (old school linux with gcc 5.4.0 20160609) or the builder dev binaries (4.2.1 Compatible Clang 3.8.0)
CC: @benesch, @windchan7 who assisted in debugging.
I'll double check reproducible binaries and start debugging.