When you manually set the GvrPointerInputModule.Pointer, the GvrPointerGraphicRaycaster doesn't clear the cachedPointerEventCamera field. Adding

if (cachedPointerEventCamera != null &&
    cachedPointerEventCamera.gameObject != GvrPointerInputModule.Pointer.PointerTransform.gameObject)
{
    cachedPointerEventCamera = null;
}

between line 175 and 176 in GvrPointerGraphicRaycaster will fix this issue.

Found using Gvr v1.150.0