[swss]: Listen for undeliverable IPinIP packets by theasianpianist · Pull Request #9348 · sonic-net/sonic-buildimage

theasianpianist · 2021-11-23T01:56:05Z

Why I did it

In dual ToR setups, it is possible for the standby to encap a server-bound packet to the active ToR when the active ToR does not have neighbor information for the packet's destination IP. In this case, the packet will be trapped to the CPU and dropped with no further action taken. This can result in blackholed traffic.

How I did it

Create a script in the orchagent docker container which listens for these trapped encapsulated packets. When such a packet is received, the script will issue a ping command to the packet's inner destination IP to start the neighbor learning process.

This script is also resilient to portchannel status changes (i.e. interface going up or down). An interface going down does not affect traffic sniffing on interfaces which are still up. When an interface comes back up, we restart the sniffer to start capturing traffic on that interface again.

How to verify it

Send an IPinIP packet to the active ToR:

<Ether  dst=94:8e:d3:04:eb:28 type=0x800 |<IP  proto=ipv6 src=10.1.0.33 dst=10.1.0.32 |<IPv6  src=2603:10b0:10b:8a8e::a56:e825 dst=fc02:1000::99 |>>>

This packet must meet the following requirements:

Destination MAC matches the ToR's device MAC (which may be different from the VLAN MAC)
The outer source IP should be the PEER_SWITCH IP defined in the PEER_SWITCH table in config DB
The outer destination IP should be the ToR's Loopback0 IP
The inner destination IP should be contained with the ToR's VLAN subnet

Verify that the inner destination IP is added to the kernel neighbor table:

admin@sonic:/var/log/swss$ ip -6 neigh
...
fc02:1000::99 dev Vlan1000  FAILED

Verify via packet capture that neighbor solicitation messages are sent by the kernel on the VLAN interface:

admin@sonic:~$ sudo tcpdump -i Vlan1000 -nnl
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on Vlan1000, link-type EN10MB (Ethernet), capture size 262144 bytes
01:40:09.581164 IP6 fc02:1000::1 > ff02::1:ff00:99: ICMP6, neighbor solicitation, who has fc02:1000::99, length 32
01:40:10.604827 IP6 fc02:1000::1 > ff02::1:ff00:99: ICMP6, neighbor solicitation, who has fc02:1000::99, length 32
01:40:11.628845 IP6 fc02:1000::1 > ff02::1:ff00:99: ICMP6, neighbor solicitation, who has fc02:1000::99, length 32

Verify that the ping command was recorded in the syslog:

Nov 23 01:40:09.576360 sonic INFO swss#ipinip_ping.py: Running command 'ping -c1 -W1 -6 fc02:1000::99'

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

dockers/docker-orchagent/ipinip_ping.py

prsunny · 2021-11-23T20:14:41Z

dockers/docker-orchagent/ipinip_ping.py

+        logger.log_notice('Starting IPinIP listener for IPs {} and {}'
+                          .format(self_ip, peer_ip))
+        while True:
+            sniff(


could you check the cpu utilization of this? especially, if have a lots of packet trapped to Lpbk0 IP?

The maximum rate that scapy can send packets from the PTF is about 170 KB/s. Sending IPinIP packets at this rate causes utilization to hover around ~2%, never exceeding ~2.6%

admin@str2-7050cx3-acs-08:~$ ps ax | grep ipinip_ping | grep -v grep 91073 pts/0 S 0:03 python3 /usr/bin/ipinip_ping.py admin@str2-7050cx3-acs-08:~$ top | grep 91073 91073 root 20 0 38704 28308 9376 S 2.3 0.2 0:03.65 python3 91073 root 20 0 38960 28364 9376 S 2.0 0.2 0:03.71 python3 91073 root 20 0 38960 28432 9376 S 1.7 0.2 0:03.76 python3 91073 root 20 0 38960 28508 9376 S 2.6 0.2 0:03.84 python3 91073 root 20 0 38960 28568 9376 S 1.7 0.2 0:03.89 python3 91073 root 20 0 38960 28628 9376 S 2.0 0.2 0:03.95 python3 91073 root 20 0 39216 28692 9376 S 2.0 0.2 0:04.01 python3 91073 root 20 0 39216 28772 9376 S 2.0 0.2 0:04.07 python3 91073 root 20 0 39216 28840 9376 S 2.0 0.2 0:04.13 python3 91073 root 20 0 39348 28904 9376 S 2.3 0.2 0:04.20 python3

- Use STATE_DB INTERFACE_TABLE to check portchannel status - Cache portchannel members from config DB - Wrap `ping` command with `timeout` for graceful exit - Verify that the tunnel type in config DB is IPinIP Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

prsunny · 2021-11-29T21:32:46Z

dockers/docker-orchagent/ipinip_ping.py

+        """
+        start = datetime.now()
+
+        while (datetime.now() - start).seconds < 60:


this should be timeout, correct?

prsunny · 2021-11-29T21:35:06Z

dockers/docker-orchagent/ipinip_ping.py

+
+        self_ip, peer_ip = self.get_ipinip_tunnel_addrs()
+        if self_ip is None or peer_ip is None:
+            logger.log_error('Could not get IPinIP tunnel addresses from '


this need not be error in single tor. Lets have it as notice

prsunny · 2021-12-01T23:01:32Z

dockers/docker-orchagent/ipinip_ping.py

@@ -0,0 +1,233 @@
+#! /usr/bin/env python3
+"""
+Adds neighbor to kernel for undeliverable IPinIP packets


Suggest renaming this to tunnel_packet_handler.py so we can extend this to Vxlan in future

- Use argument instead of hardcoded value for timeout - Reduce some logging severity Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

theasianpianist · 2021-12-06T23:42:09Z

/Azp run

azure-pipelines · 2021-12-06T23:42:13Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

theasianpianist · 2021-12-08T20:54:06Z

/Azp run all

azure-pipelines · 2021-12-08T20:54:11Z

No pipelines are associated with this pull request.

prsunny · 2021-12-08T23:01:39Z

dockers/docker-orchagent/tunnel_packet_handler.py

+        pc_index_map = self.get_portchannel_index_mapping()
+        for msg in messages:
+            if msg['index'] in pc_index_map:
+                if msg['state'] == 'up':


Lets check for interface flap cases as a future enhancement.

theasianpianist · 2021-12-09T22:11:28Z

/Azp run

azure-pipelines · 2021-12-09T22:11:33Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

theasianpianist · 2021-12-09T22:13:02Z

/Azp run Azure.sonic-buildimage

azure-pipelines · 2021-12-09T22:13:14Z

Azure Pipelines successfully started running 1 pipeline(s).

- Create a script in the orchagent docker container which listens for these encapsulated packets which are trapped to CPU (indicating that they cannot be routed/no neighbor info exists for the inner packet). When such a packet is received, the script will issue a ping command to the packet's inner destination IP to start the neighbor learning process. - This script is also resilient to portchannel status changes (i.e. interface going up or down). An interface going down does not affect traffic sniffing on interfaces which are still up. When an interface comes back up, we restart the sniffer to start capturing traffic on that interface again.

theasianpianist added 3 commits November 19, 2021 01:56

[swss]: Add ipinip_ping utility

7e233ff

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

[swss]: Make ipinip_ping.py executable

864718f

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

[swss]: Convert ipinip_ping to class

51a2a15

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

theasianpianist added the Request for 202012 Branch label Nov 23, 2021

theasianpianist requested review from lguohan and prsunny November 23, 2021 01:56

theasianpianist added 2 commits November 23, 2021 01:57

[swss]: Update ipinip_ping start order and comments

1529001

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

[swss]: Fix dockerfile copy issue

5d4ca6a

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

prsunny reviewed Nov 23, 2021

View reviewed changes

theasianpianist added 2 commits November 24, 2021 19:51

[swss]: Account for IPv4 inner packet in ipinip_ping

0a2ee7d

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

prsunny reviewed Nov 29, 2021

View reviewed changes

prsunny reviewed Dec 1, 2021

View reviewed changes

theasianpianist added 2 commits December 2, 2021 21:26

[swss]: Address review comments

60c1dd7

- Use argument instead of hardcoded value for timeout - Reduce some logging severity Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

[swss]: Rename ipinip_ping to tunnel_packet_handler

8d5f7ba

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

prsunny previously approved these changes Dec 2, 2021

View reviewed changes

[swss] Account for PC status changes in tunnel pkt handler

8838f2e

Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

theasianpianist dismissed prsunny’s stale review via 8838f2e December 4, 2021 00:06

prsunny reviewed Dec 8, 2021

View reviewed changes

prsunny approved these changes Dec 8, 2021

View reviewed changes

theasianpianist merged commit 7bd0a2a into sonic-net:master Dec 14, 2021

qiluo-msft added the Included in 202012 Branch label Jan 5, 2022

Conversation

theasianpianist commented Nov 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

prsunny Nov 23, 2021

Choose a reason for hiding this comment

Uh oh!

theasianpianist Nov 24, 2021

Choose a reason for hiding this comment

Uh oh!

prsunny Nov 29, 2021

Choose a reason for hiding this comment

Uh oh!

prsunny Nov 29, 2021

Choose a reason for hiding this comment

Uh oh!

prsunny Dec 1, 2021

Choose a reason for hiding this comment

Uh oh!

theasianpianist commented Dec 6, 2021

Uh oh!

azure-pipelines bot commented Dec 6, 2021

Uh oh!

theasianpianist commented Dec 8, 2021

Uh oh!

azure-pipelines bot commented Dec 8, 2021

Uh oh!

prsunny Dec 8, 2021

Choose a reason for hiding this comment

Uh oh!

theasianpianist commented Dec 9, 2021

Uh oh!

azure-pipelines bot commented Dec 9, 2021

Uh oh!

theasianpianist commented Dec 9, 2021

Uh oh!

azure-pipelines bot commented Dec 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

theasianpianist commented Nov 23, 2021 •

edited

Loading