From slack:

I have a progress in debugging. I confirmed the issue goes away by replacing igl::rigid_alignment() with [alternate solver]. It looks like the translation of the igl one blows up in some cases. I haven’t had a chance to dump out the exact data that blows up.
This is an example input from a real scene that blew up the translation output from rigid_alignment(VXRT, P, N, Rup, Tup). Apparently this happens when all the
closest point searches end up at the same point. It seems very easy to recreate this case by having all the same rows in P and N, with arbitrary VXRT.
And I realize [alternate solver] is not a drop-in replacement of igl::rigid_alignment() as the former is a point-point alignment, the latter is a point-plane alignment.
For now, a workaround is optimizing just the translation before calling igl::rigid_alignment(). This seems OK in practice but is not really solving the fundamental issue.

Archive.zip

This change switches ldlt (cholesky) for rank-revealing qr. If the rank is too low, then a weight is increased on a point-to-point regularization term.