I think I discovered the cause of the "phantom hit" bug (where Laserdist claims to have measured a hit very very close to the laser (usually about 0.6 meters) when there is nothing there to hit), and this should fix that cause.

Today I finally had a repeatable case where the problem happened all the time with certainty, so I could play with it and find out what happened.

I was barking up the wrong tree all this time. I assumed the Unity raycast was hitting a polygon drawn on a layer I should be masking off, but I couldn't figure out which layer that is. Maybe the map view or something, but every time I looked it looked like I was masking off the correct layers. When I figured out how to show which layer the raycast hit came from, it always showed "", indicating it wasn't coming from the actual raycast at all.

The false positive had to be coming from the PQS fallback algorithm I wrote that it resorts to when there is no raycast hit.

Background on the PQS fallback system: If you aim the laser at a mountain a few km away, the Physics.Raycast will claim there's nothing there because it's outside the range where the game populates colliders for terrain polygons. So I use a fallback when there's no Physics.Raycast hit - I instead use an ugly iterative numeric method that plugs sample points along the line into the PQS terrain height function to find a "slice" of the line where the point on one side is above the terrain and the point on the other side is below the terrain. Then it recursively slices this slice into smaller slices and checks again, etc, till it finds the spot within an acceptable epsilon.

So why was it returning a false positive?

(see diagram below) Because the physical terrain polygons that make a course-grain approximation of the smoother PQS ideal terrain often end up being underground as far as the PQS terrain height claims the altitude should be. If the terrain is course-grain enough, and the PQS terrain is a sharp enough curve, the error in the straight-line approximations can be large enough to fit a vessel underneath what the PQS terrain claims is the ideal terrain height. Then a LaserDist part on that vessel can emit a laser that starts off already underground as far as the PQS system thinks of it.

The phantom hits were coming from the spot where the laser line first emits OUT from under the PQS hypothetical terrain, not the spot where it dives down INTO the terrain.

The fix was to put a few guard checks in place to help prevent this:

1 - Don't count a hit until after at least one sample point along the line was seen that is above the ground.
2 - If a hit is returned from the PQS fallback algorithm that is less than 2km away, ignore it and call it a miss. Within 2km it shouldn't have to resort to using the PQS fallback algorithm - the polygons within 2km should be populated with proper colliders - so if it fails to find a true raycast hit within 2km, then any PQS hits it finds within 2km can't possibly be real.