Fixes WB-20957.

Retries checksum errors when syncing TensorBoard files. Previously, the integration would stop syncing tfevents files upon encountering these errors; now it logs a warning and retries later (the default delay is 5 seconds).

See this list of events:

tensorboard: failed reading next event: tensorboard: unexpected CRC-32C checksum for event. Expected: 24, got: 2894970002

In almost every single one, the Expected number (the checksum read from the file rather than calculated from the data) is small: usually 0, but in some instances 24, 12869, 2878429. The checksum is stored in little-endian format, so a simple explanation would be that its final (most significant) bytes are not yet written.

Unfortunately for that explanation, there are plenty of counter-examples, like this event:

(big endian bytes)
5F 6C 2F 73 (1600925555) - checksum read from the file
B2 1A EF 6D (2988109677) - checksum computed from the payload

A way to account for this is that the checksum and payload may be written independently, possibly using a mechanism like mmap. When mmap is used, we can't assume the data in the file is final: the writer may have reserved space for the event but not yet written all its bytes. When we see an invalid checksum, we should reread the data.

But still, this doesn't explain all the checksum errors we see. In some cases, the same "expected" and "actual" checksums appear in multiple events spaced hours apart, so there is probably some kind of data corruption.