Skip to content

Cilium non-deterministically classifies CIDR policy matches for range with node IPs #16308

Closed
#25298
Closed
Cilium non-deterministically classifies CIDR policy matches for range with node IPs#16308
#25298
Assignees
asauber
Labels
area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.pinnedThese issues are not marked stale by our issue bot.sig/policyImpacts whether traffic is allowed or denied based on user-defined policies.
@olemarkus

Description

@olemarkus
olemarkus
opened on May 25, 2021

Bug report

General Information

  • Cilium version: cilium 1.9.4
  • Kernel version: Linux ip-10-29-109-25 5.4.0-1045-aws Adding labels to ui #47-Ubuntu SMP Tue Apr 13 07:02:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • Orchestration system version in use: kubernetes v1.20.6

How to reproduce the issue

Run node-local-dns --setupiptables=false and kubelet --cluster-dns=169.254.20.10

Add a CNP that looks like this:

spec:
  egress:
  - toCIDR:
    - 169.254.20.10/32
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
  endpointSelector: {}

On first cilium run, cilium will deny traffic to node-local-dns. If the pod is deleted, the second cilium run will accept traffic.

First cilium run

cilium ip list:

169.254.20.10/32    host

cilium monitor -t policy-verdict:

Policy verdict log: flow 0xaaa04ad local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:53076 -> 169.254.20.10:53 udp
Policy verdict log: flow 0xaaa04ad local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:53076 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x0 local EP ID 481, remote ID 17956, proto 6, ingress, action deny, match none, 100.71.139.54:45062 -> 100.71.181.177:6943 tcp SYN
Policy verdict log: flow 0xaaa04ad local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:53076 -> 169.254.20.10:53 udp
Policy verdict log: flow 0xaaa04ad local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:53076 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x897b78c4 local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:56843 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x897b78c4 local EP ID 1762, remote ID 1, proto 17, egress, action deny, match none, 100.71.181.66:56843 -> 169.254.20.10:53 udp

Second cilium run

169.254.20.10/32     cidr:169.254.0.0/16                                                                                       
                     cidr:169.254.0.0/19                                                                                       
                     cidr:169.254.20.10/31                                                                                     
                     cidr:168.0.0.0/5                                                                                          
                     cidr:168.0.0.0/7                                                                                          
                     cidr:169.254.0.0/17                                                                                       
                     cidr:169.254.20.0/23                                                                                      
                     reserved:world                                                                                            
                     cidr:169.254.20.0/22                                                                                      
                     cidr:169.254.20.10/32                                                                                     
                     cidr:169.254.20.0/25                                                                                      
                     cidr:169.254.20.0/27                                                                                      
                     cidr:169.0.0.0/8                                                                                          
                     cidr:169.128.0.0/9                                                                                        
                     cidr:169.254.0.0/15                                                                                       
                     cidr:169.224.0.0/11                                                                                       
                     cidr:168.0.0.0/6                                                                                          
                     cidr:169.254.20.0/26                                                                                      
                     cidr:160.0.0.0/3                                                                                          
                     cidr:169.254.16.0/20                                                                                      
                     cidr:0.0.0.0/0                                                                                            
                     cidr:169.254.16.0/21                                                                                      
                     cidr:128.0.0.0/2                                                                                          
                     cidr:169.248.0.0/13                                                                                       
                     cidr:169.192.0.0/10                                                                                       
                     cidr:169.254.20.8/30                                                                                      
                     cidr:169.254.0.0/18                                                                                       
                     cidr:160.0.0.0/4                                                                                          
                     cidr:169.254.20.0/28                                                                                      
                     cidr:169.254.20.8/29                                                                                      
                     cidr:169.240.0.0/12                                                                                       
                     cidr:169.252.0.0/14                                                                                       
                     cidr:169.254.20.0/24                                                                                      
                     cidr:128.0.0.0/1

cilium monitor -t policy-verdict:

Policy verdict log: flow 0x5f42acab local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:38685 -> 169.254.20.10:53 udp
Policy verdict log: flow 0xbe339248 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:54033 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x3a516849 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:40686 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x656356b6 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:41152 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x978580ec local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:53109 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x45c1e6c1 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:50593 -> 169.254.20.10:53 udp
Policy verdict log: flow 0x9cd850d5 local EP ID 1109, remote ID 16777221, proto 17, egress, action allow, match L3-L4, 100.71.139.123:37507 -> 169.254.20.10:53 udp

Slack thread https://cilium.slack.com/archives/C1MATJ5U5/p1621963201289500

Metadata

Metadata

Assignees

Labels

area/datapathImpacts bpf/ or low-level forwarding details, including map management and monitor messages.kind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and next steps.pinnedThese issues are not marked stale by our issue bot.sig/policyImpacts whether traffic is allowed or denied based on user-defined policies.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions