The only thing that gives out 3D ray information from a 2D view is depth images with a defined meter right now. We fill in the image plane distance otherwise.
Should make the flow of this depth information more principled to account for 3D objects. Related to #1581