When running in tunneling mode with a kube-apiserver in the cluster, the
node with the kube-apiserver will transmit packets towards remote pods
with identity 6 (remote-node).

The recipient node will receive the packet and trust the identity as
"remote-node" and not "kube-apiserver". If the receiving pod has a
policy to only allow "from kube-apiserver" then this policy would not
work, and thus drop the packet.

To fix this, we switch the source identity in bpf_overlay from
"remote-node" to the identity of the destination IP via the ipcache,
which we expect to be the "kube-apiserver" identity. If the ipcache
entry doesn't exist or the identity in the ipcache is "remote-node"
already, then this is a no-op (besides the extra ipcache lookup).

Going forward, there are two future changes that are needed in order:

1) A subsequent change to make any packet associated with the
   kube-apiserver IP have ID 7 (rather than ID 6; without needing the
   logic added by this commit).
2) Once v1.12.y is released, another subsequent change to remove this
   specific code added in this commit because we can expect all of this
   traffic to be transmitted with "kube-apiserver" identity (7),
   **after** (1) is implemented.

For a hitless upgrade, we would only support a path from Cilium v1.11.x
(this commit would need to be backported to v1.11.x) to v1.12.y. This is
because only v1.11.x or later can successfully receive this traffic and
handle policy correctly, as guaranteed by the above.

Co-authored-by: Joe Stringer <joe@cilium.io>
Signed-off-by: Chris Tarazi <chris@isovalent.com>

Additional sanity check coverage that the kube-apiserver policy matching
feature still works with a duplicate policy added and the previous
removed.

Updates: cilium#17829

Signed-off-by: Chris Tarazi <chris@isovalent.com>