Commit f25d8b9 ("bpf: Preserve source identity for hairpin via stack")
recently broke BPF NodePort since the use of MARK_MAGIC_SNAT_DONE is now in
conflict/overlapping with MARK_MAGIC_IDENTITY.

In tc egress hooks of bpf_{netdev,overlay} we have a check whether the SNAT
is not needed via if ((ctx->mark & MARK_MAGIC_SNAT_DONE) == MARK_MAGIC_SNAT_DONE).

MARK_MAGIC_SNAT_DONE is 0x0500 whereas MARK_MAGIC_IDENTITY is 0x0F00. So far
it was never a problem since MARK_MAGIC_IDENTITY was not used in a path where
MARK_MAGIC_SNAT_DONE was tested until this changed via f25d8b9 where now
SNAT was skipped for Pod traffic.

As a result, we've seen various flakes in SNAT or Hybrid NodePort setups mostly
on the tftp/UDP tests. Reverting f25d8b9 confirmed to fix these flakes. As
a workaround/fix, remap MARK_MAGIC_SNAT_DONE to 0x1500, so that setting the
MARK_MAGIC_IDENTITY will get a mismatch on above mentioned test and hence we'll
end up doing the SNAT.

This reaches into MARK_MAGIC_KEY_ID bit space, but the agent today cannot be
configured to run with both BPF NodePort and IPSec at the same time.

Fixes: #10942
Fixes: f25d8b9 ("bpf: Preserve source identity for hairpin via stack")
Reported-by: Martynas Pumputis <m@lambda.lt>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>