The white part of the ray should hit the depth image, but it doesn't because it seems we're taking the distance for the ray length directly without accounting for perspective

Meaning that in the middle of the image it works out: