Skip to content

Fix DNS egress L7 policy in ENI mode#8837

Merged
ianvernon merged 1 commit intocilium:masterfrom
joestringer:submit/eni-proxy
Aug 10, 2019
Merged

Fix DNS egress L7 policy in ENI mode#8837
ianvernon merged 1 commit intocilium:masterfrom
joestringer:submit/eni-proxy

Conversation

@joestringer
Copy link
Copy Markdown
Member

@joestringer joestringer commented Aug 8, 2019 •
edited
Loading

ENI mode was disabling L7 proxy entirely previously due to an overeager check in the datapath to avoid redirecting to the proxy in cases where the endpoint devices are IPVLAN devices.

These fixes enable the following cases in ENI mode:

  • Egress L7 policy plus no ingress policy

Fixes: #8813

Future work:

This change is Reviewable

@joestringer joestringer added pending-review area/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. labels Aug 8, 2019
@joestringer joestringer requested review from a team August 8, 2019 04:57
@joestringer
Copy link
Copy Markdown
Member Author

joestringer commented Aug 8, 2019

test-me-please

@coveralls
Copy link
Copy Markdown

coveralls commented Aug 8, 2019 •
edited
Loading

Coverage Status

Coverage decreased (-0.009%) to 44.169% when pulling dea7a69 on joestringer:submit/eni-proxy into a234302 on cilium:master.

borkmann
borkmann reviewed Aug 8, 2019
View reviewed changes
Copy link
Copy Markdown
Member

@borkmann borkmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't ENI mode simply define ENABLE_HOST_REDIRECT then?
L7 with ipvlan is not supported (https://cilium.readthedocs.io/en/stable/gettingstarted/ipvlan/) at this point.

@tgraf
Copy link
Copy Markdown
Contributor

tgraf commented Aug 8, 2019

@joestringer I remember that for the lyft-cni chaining, it was required to specifically route via cilium_host in order to get the redirect to work. It's interesting that this is no longer the case apparently. Could also be specific to kernel versions. I remember @jrfastab saw similar issues related to the encryption work.

@joestringer
Copy link
Copy Markdown
Member Author

joestringer commented Aug 9, 2019 •
edited
Loading

Enabling host redirect fixes DNS egress proxy iirc, but had other problems. Long and short is that right now if I just enable host redirect it causes ingress proxy at destination to receive SYN, reply with SYN/ACK, then fail to receive the ACK (and retransmit SYN/ACK ad infinitum). SYN/ACK reaches source node which responds with ACK, which arrives to eth1 of destination but I haven't been able to trace it further than that yet.

EDIT: Here's a rough sense of the netfilter stack paths for this configuration:

[240175.580516] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=253 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                               
[240175.594056] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=253 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                               
[240175.607625] CILIUM_PRE_nat:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=253 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                               
[240175.621522] CILIUM_FORWARD:     IN=eth1 OUT=lxc61b3fb1d7642 MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=252 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                
[240175.637058] CILIUM_POST_mangle: IN= OUT=lxc61b3fb1d7642 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=252 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                                                                  
[240175.649602] CILIUM_POST_nat:    IN= OUT=lxc61b3fb1d7642 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=252 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0                                                                                  
[240175.662060] CILIUM_PRE_raw:     IN=cilium_host OUT= MAC=66:70:85:22:a1:91:aa:6a:86:da:04:ac:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=251 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0 MARK=0xe83b0200                        
[240175.678050] CILIUM_PRE_mangle:  IN=cilium_host OUT= MAC=66:70:85:22:a1:91:aa:6a:86:da:04:ac:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=251 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0 MARK=0xe83b0200                        
[240175.694148] INPUT:              IN=cilium_host OUT= MAC=66:70:85:22:a1:91:aa:6a:86:da:04:ac:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=60 TOS=0x00 PREC=0x00 TTL=251 ID=65417 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=26883 RES=0x00 SYN URGP=0 MARK=0x200                             
[240175.709985] CILIUM_OUTPUT_raw:  IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240175.722579] OUTPUT:             IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240175.735015] CILIUM_POST_mangle: IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240175.747619] CILIUM_PRE_raw:     IN=cilium_net OUT= MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                         
[240175.762845] CILIUM_PRE_mangle:  IN=cilium_net OUT= MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                         
[240175.778255] CILIUM_FORWARD:     IN=cilium_net OUT=eth1 MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                     
[240175.793722] CILIUM_POST_mangle: IN= OUT=eth1 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                                                                             
[240175.807333] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=52 TOS=0x00 PREC=0x00 TTL=253 ID=65418 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK URGP=0                                                 
[240175.820700] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=52 TOS=0x00 PREC=0x00 TTL=253 ID=65418 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK URGP=0                                                 
[240175.834169] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=148 TOS=0x00 PREC=0x00 TTL=253 ID=65419 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK PSH URGP=0                                            
[240175.847866] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=148 TOS=0x00 PREC=0x00 TTL=253 ID=65419 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK PSH URGP=0                                            
[240176.464183] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=148 TOS=0x00 PREC=0x00 TTL=253 ID=65420 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK PSH URGP=0                                            
[240176.478037] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=148 TOS=0x00 PREC=0x00 TTL=253 ID=65420 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK PSH URGP=0                                            
[240176.730940] CILIUM_OUTPUT_raw:  IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240176.743434] OUTPUT:             IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240176.756019] CILIUM_POST_mangle: IN= OUT=cilium_host SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0 MARK=0xa00                                                                           
[240176.768717] CILIUM_PRE_raw:     IN=cilium_net OUT= MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                         
[240176.784031] CILIUM_PRE_mangle:  IN=cilium_net OUT= MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=255 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                         
[240176.799456] CILIUM_FORWARD:     IN=cilium_net OUT=eth1 MAC=8a:44:05:29:fe:05:66:70:85:22:a1:91:08:00 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                     
[240176.814885] CILIUM_POST_mangle: IN= OUT=eth1 SRC=192.168.47.21 DST=192.168.94.61 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=0 DF PROTO=TCP SPT=80 DPT=34388 WINDOW=26847 RES=0x00 ACK SYN URGP=0                                                                                             
[240176.828098] CILIUM_PRE_raw:     IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=52 TOS=0x00 PREC=0x00 TTL=253 ID=65421 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK URGP=0                                                 
[240176.947137] CILIUM_PRE_mangle:  IN=eth1 OUT= MAC=0a:8a:c1:c9:2f:00:0a:44:02:cb:2a:5e:08:00 SRC=192.168.94.61 DST=192.168.47.21 LEN=52 TOS=0x00 PREC=0x00 TTL=253 ID=65421 DF PROTO=TCP SPT=34388 DPT=80 WINDOW=211 RES=0x00 ACK URGP=0

@joestringer
datapath: Enable host redirect in ENI mode
dea7a69 
Signed-off-by: Joe Stringer <joe@cilium.io>
@joestringer joestringer changed the title Fix some L7 policy cases in ENI mode Fix DNS egress L7 policy in ENI mode Aug 9, 2019
@joestringer
Copy link
Copy Markdown
Member Author

joestringer commented Aug 9, 2019

I've updated the PR to simplify it, this is easier to justify and it fixes the specific case that was reported.

Getting ingress policy to work in ENI mode across multiple nodes will require additional work.

@joestringer
Copy link
Copy Markdown
Member Author

joestringer commented Aug 9, 2019

test-me-please

@ianvernon ianvernon merged commit a96d7f4 into cilium:master Aug 10, 2019
@joestringer joestringer deleted the submit/eni-proxy branch August 10, 2019 16:27
@ianvernon ianvernon mentioned this pull request Aug 12, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@borkmann borkmann borkmann approved these changes

+1 more reviewer

@ianvernon ianvernon ianvernon approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

DSN resolution does not work with Cillium agent in ENI IPAM allocation mode

5 participants

@joestringer @coveralls @tgraf @borkmann @ianvernon