Ceph » RADOS

I'm the one who was bit by this, and it appears to have been caused by the fact that I apparently didn't investigate Blaum-Roth parameters sufficiently, and used the default value of w=7.

Page 29 of the Jerasure manual (https://github.com/ceph/jerasure/blob/de1739cc8483696506829b52e7fda4f6bb195e6a/Manual.pdf) indicates that w+1 must be prime, but the default value therefore results in an invalid configuration.

Back in #6754 (12 years ago!), an attempt to validate this was made, but the practical effect was quickly reverted in #6572 (which, despite the numbering, came after the first change).

I understand the desire for backwards compatibility (and certainly it is better to have some data accessible than none, if Ceph refuses to start), but the default value for w here causes data corruption (with low probability on any given EC object, but high probability overall for large enough datasets).

If at all possible, I would suggest changing the default value of w to 6, and, ideally, producing a noticeable warning about w=7 (perhaps as a health issue in ceph status): w=7 almost certainly means that data corruption is a real possibility if a disk dies, or rebalancing occurs.

I suspect that changing the default is as simple as adding back in the default value that was removed in https://github.com/ceph/ceph/pull/2556 , but I'd like someone more experienced in Ceph's EC code to verify that doing so won't change the expected values for existing pools that were using the defaults. (Adding a status warning is likely more complicated, and I don't know how to get from the check_w function to a Ceph status entry.)

As a note for anyone reading this issue who finds themselves in a similar situation, I wrote some rough code to recover data from my blaum_roth w=7 pool by exhaustively combining all EC shards and taking the correct entries. It is at https://github.com/aschmitz/ceph-ec-recovery , but may or may not work in all situations, so I'm mostly providing it as a pointer rather than a complete solution.

Oops, I typo'd that second issue number, which explains why I thought they were out of order. The issue reverting the validation and fix to blaum_roth defaults was actually #9572.

Thanks for the updates @aschmitz - really interesting read on github and I'm glad you managed to recover the data.

We should change the default value for w when creating a new blaum_roth profile to be 6 instead of 7.

I think the CLI already returns an error message when creating a new blaum_roth profile if you specify a w value where w + 1 is not a prime number. We should check this is true and add this policing if it doesn't already exist.

We should also add a new warning to the health monitor to warn users who have EC pools using a blaum_roth profile if w + 1 is not a prime number. Something like "An EC pool is using the blaum_roth technique and w + 1 is not a prime number. This can result in data corruption."

Let's start from just changing the default.

PR in draft state; PR under review (2nd round of review needed)

Scrub note: undergoing further review

scrub mote: went into QA.

scrub note: awaits QA.

Still in QA.