In the following setup, mounting a CephFS/RBD is not possible:

• Cilium >= 1.6.5
• Rook 1.2.2 (all versions with CSI enabled)
• CoreOS 2303 (most likely older versions affected as well)
• Cilium Config enable-host-reachable-services: true

This is caused by cilium optimizing the ServiceIP <-> PodIP NAT step by hooking into the connect(), sendmsg(), recvmsg() syscalls using eBPF. Unfortunately, there is no way to hook into getpeername() which is used by ceph to verify the remote client. There is work in progress fixing the issue in net-next by @borkmann.

Until then, it's required to disable 'host-reachable-services'. This is also documented on the cilium wiki: https://cilium.readthedocs.io/en/stable/gettingstarted/host-services/#limitations

When hitting the problem, the kernel log looks like this:

[1141671.083024] libceph: wrong peer, want $podip:6789/0, got $serviceip:6789/0
[1141671.084930] libceph: mon0 $podip:6789 wrong peer at address

cc @galexrt