While adding support for IPv6 to the enterprise a4x2 topology (static routing + BFD), I found an issue with IPv6 BFD such that the peer (FRR) rejects BFD control packets sent by mgd.
BFD debugs on the FRR side showed that the packets were rejected due to the IPv6 Hop Limit not being set to 255, which is mandatory per RFC 5881.
2026-03-04 19:25:53 [DEBG] bfdd: [YA0Q5-C0BPV] control-packet: invalid TTL: 60 expected 255 [mhop:no peer:fd00:101::6 local:fd00:101::5 port:4]
2026-03-04 19:25:53 [DEBG] bfdd: [YA0Q5-C0BPV] control-packet: invalid TTL: 60 expected 255 [mhop:no peer:fd00:101::2 local:fd00:101::1 port:2]
2026-03-04 19:25:54 [DEBG] bfdd: [YA0Q5-C0BPV] control-packet: invalid TTL: 60 expected 255 [mhop:no peer:fd00:101::a local:fd00:101::9 port:3]
2026-03-04 19:25:54 [DEBG] bfdd: [YA0Q5-C0BPV] control-packet: invalid TTL: 60 expected 255 [mhop:no peer:fd00:101::e local:fd00:101::d port:5]
2026-03-04 19:25:54 [DEBG] bfdd: [YA0Q5-C0BPV] control-packet: invalid TTL: 60 expected 255 [mhop:no peer:fd00:101::6 local:fd00:101::5 port:4]
2026-03-04 19:25:54 [DEBG] bfdd: [YA0Q5-C0BPV] control-packet: invalid TTL: 60 expected 255 [mhop:no peer:fd00:101::2 local:fd00:101::1 port:2]
2026-03-04 19:25:55 [DEBG] bfdd: [YA0Q5-C0BPV] control-packet: invalid TTL: 60 expected 255 [mhop:no peer:fd00:101::a local:fd00:101::9 port:3]
root@cfw:~# tcpdump -eni enp0s9 udp port 3784 or udp port 3785 -vvv
tcpdump: listening on enp0s9, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:55:17.565593 a8:40:25:54:c2:8d > a8:40:25:00:00:10, ethertype IPv6 (0x86dd), length 86: (hlim 60, next-header UDP (17) payload length: 32) fd00:101::a.49153 > fd00:101::9.3784: [udp sum ok] BFDv1, length: 24
Control, State Init, Flags: [none], Diagnostic: No Diagnostic (0x00)
Detection Timer Multiplier: 3 (3000 ms Detection time), BFD Length: 24
My Discriminator: 0x4ec16a0b, Your Discriminator: 0x2fd374cf
Desired min Tx Interval: 1000 ms
Required min Rx Interval: 1000 ms
Required min Echo Interval: 0 ms
18:55:17.707340 a8:40:25:54:c2:8d > a8:40:25:00:00:10, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 255, id 8987, offset 0, flags [none], proto UDP (17), length 52)
198.51.101.10.49155 > 198.51.101.9.3784: [udp sum ok] BFDv1, length: 24
Control, State Up, Flags: [none], Diagnostic: No Diagnostic (0x00)
Detection Timer Multiplier: 3 (3000 ms Detection time), BFD Length: 24
My Discriminator: 0xc2714021, Your Discriminator: 0x4d2dc59a
Desired min Tx Interval: 1000 ms
Required min Rx Interval: 1000 ms
Required min Echo Interval: 0 ms
18:55:17.825937 a8:40:25:00:00:10 > a8:40:25:54:c2:8d, ethertype IPv6 (0x86dd), length 86: (class 0xc0, flowlabel 0xfc1ea, hlim 255, next-header UDP (17) payload length: 32) fd00:101::9.49157 > fd00:101::a.3784: [udp sum ok] BFDv1, length: 24
Control, State Down, Flags: [none], Diagnostic: Administratively Down (0x07)
Detection Timer Multiplier: 3 (3000 ms Detection time), BFD Length: 24
My Discriminator: 0x2fd374cf, Your Discriminator: 0x00000000
Desired min Tx Interval: 1000 ms
Required min Rx Interval: 1000 ms
Required min Echo Interval: 50 ms
18:55:17.914973 a8:40:25:00:00:10 > a8:40:25:54:c2:8d, ethertype IPv4 (0x0800), length 66: (tos 0xc0, ttl 255, id 54401, offset 0, flags [DF], proto UDP (17), length 52)
198.51.101.9.49153 > 198.51.101.10.3784: [udp sum ok] BFDv1, length: 24
Control, State Up, Flags: [none], Diagnostic: No Diagnostic (0x00)
Detection Timer Multiplier: 3 (900 ms Detection time), BFD Length: 24
My Discriminator: 0x4d2dc59a, Your Discriminator: 0xc2714021
Desired min Tx Interval: 300 ms
Required min Rx Interval: 300 ms
Required min Echo Interval: 50 ms
18:55:18.008442 a8:40:25:54:c2:8d > a8:40:25:00:00:10, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 255, id 8989, offset 0, flags [none], proto UDP (17), length 52)
198.51.101.10.49155 > 198.51.101.9.3784: [udp sum ok] BFDv1, length: 24
Control, State Up, Flags: [none], Diagnostic: No Diagnostic (0x00)
Detection Timer Multiplier: 3 (3000 ms Detection time), BFD Length: 24
My Discriminator: 0xc2714021, Your Discriminator: 0x4d2dc59a
Desired min Tx Interval: 1000 ms
Required min Rx Interval: 1000 ms
Required min Echo Interval: 0 ms
18:55:18.309464 a8:40:25:54:c2:8d > a8:40:25:00:00:10, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 255, id 8991, offset 0, flags [none], proto UDP (17), length 52)
198.51.101.10.49155 > 198.51.101.9.3784: [udp sum ok] BFDv1, length: 24
Control, State Up, Flags: [none], Diagnostic: No Diagnostic (0x00)
Detection Timer Multiplier: 3 (3000 ms Detection time), BFD Length: 24
My Discriminator: 0xc2714021, Your Discriminator: 0x4d2dc59a
Desired min Tx Interval: 1000 ms
Required min Rx Interval: 1000 ms
Required min Echo Interval: 0 ms
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
root@cfw:~#
Interestingly, we do seem to set the IPv4 TTL to 255 for control packets we send (also shown in the tcpdump above).
After spending some time in the codebase, we don't explicitly set the TTL or Hop Limit at all... so it seems like Illumos may just be defaulting to TTL 255 and HL 60, allowing IPv4 to work by happenstance.
While adding support for IPv6 to the enterprise a4x2 topology (static routing + BFD), I found an issue with IPv6 BFD such that the peer (FRR) rejects BFD control packets sent by mgd.
BFD debugs on the FRR side showed that the packets were rejected due to the IPv6 Hop Limit not being set to 255, which is mandatory per RFC 5881.
tcpdump confirmed the same thing:
Interestingly, we do seem to set the IPv4 TTL to 255 for control packets we send (also shown in the tcpdump above).
After spending some time in the codebase, we don't explicitly set the TTL or Hop Limit at all... so it seems like Illumos may just be defaulting to TTL 255 and HL 60, allowing IPv4 to work by happenstance.