{"id":4861,"date":"2021-05-13T22:49:07","date_gmt":"2021-05-13T22:49:07","guid":{"rendered":"https:\/\/karneliuk.com\/?p=4861"},"modified":"2021-05-13T22:49:09","modified_gmt":"2021-05-13T22:49:09","slug":"sp-part-9-health-check-of-segment-routing-traffic-engineering-sr-te-tunnels-with-seamless-bfd-in-nokia-sr-os","status":"publish","type":"post","link":"https:\/\/karneliuk.com\/2021\/05\/sp-part-9-health-check-of-segment-routing-traffic-engineering-sr-te-tunnels-with-seamless-bfd-in-nokia-sr-os\/","title":{"rendered":"SP. Part 9. Health check of Segment Routing Traffic Engineering (SR-TE) tunnels with seamless BFD in Nokia SR OS"},"content":{"rendered":"\n

Hello my friend,<\/p>\n\n\n\n

For a long time we haven’t posted blogs about pure network technologies. However, recently we were working on some interesting use case, which so far is not yet covered at a level of the working details nowhere in the internet. As such, we decided to share with you our findings and working details.<\/p>\n\n\n\n\n\n\n\n

\n\n
1
2
3
4
5
<\/div><\/td>
No part of this blogpost could be reproduced, stored in a 
\nretrieval system, or transmitted in any form or by any 
\nmeans, electronic, mechanical or photocopying, recording, 
\nor otherwise, for commercial purposes without the 
\nprior permission of the author.<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
How automation can help with SR-TE in SP network? <\/h2>\n\n\n\n
In some (big) networks the BGP-SR-TE is a good signal the SR-TE policies, so that the PE routers can build the SR-TE tunnels without the need to configure them locally. However, the BGP-SR-TE requires a full pledged SDN controller, so that you can generate the SR-TE policy in the backend using some UI\/API and send the policies down to the network elements using the BGP. If you want to have a somewhat simpler setup, you may need need to deploy the tunnels manually. In this case, the automation is your closest ally as it can take the burden of the configuration generation and rolling it out to your devices.<\/p>\n\n\n
\n
\"\"\/<\/a><\/p><\/div>\n\n\n\n\n
In our trainings, the Live Network Automation Training (10 weeks)<\/a> and Automation with Nornir (2 weeks)<\/a>, we explore a lot of real use cases, where the automation helps you to validate the state of you network and change it if necessary. You will learn the whole spectre of the automation approaches starting from the text-based automation used in the hyper scalers with the full configuration templated till the model-driven automation with NETCONF\/RESTCONF\/gNMI loaded with YANG data models for Cisco, Nokia, Arista and Cumulus. This knowledge comes with a lot of different exercises with direct console as well as Ansible\/Python and Bash scripts. On top to that, you learn a lot of infrastructure skills such as building and managing Linux, KVM and Docker. <\/p>\n\n\n\n
Moreover, we are running our trainings for 2 years already and constantly adopting them to the changes happening in the automation world. Master your automation skills with us.<\/p>\n\n\n\n
Start your automation training today.<\/a><\/p><\/blockquote>\n\n\n\n
Brief description<\/h2>\n\n\n\n
The Segment Routing Traffic Engineering (SR-TE) tunnels are the important building blocks of the modern Service Provider networks and\/or Data Centre DCIs. Long ago we already covered the SR-TE tunnels in Nokia SR and Cisco IOS XR based network devices<\/a>. One the problem we were not able solve that time was the absence of the mechanism on the PE router to check if the tunnel is alive or not. See the following picture for better understanding.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The SR-TE tunnel by default doesn’t have a built-in mechanism to check the connectivity end-to-end, as it is just being computed at the head end router and installed in RIB. Looking into the example above, say, the SR1 has 2 SR-TE tunnels to SR4, which have path diversity and computed using the SR Adj SIDs. If link SR2 (1\/1\/1) – SR3 (1\/1\/1) is down, the SR1 doesn’t invalidate the red SR-TE tunnel and sends traffic to SR2 for it, whereas SR2 starts dropping the, as the the corresponding Adj SID points to the interface, which is down. <\/p>\n\n\n\n
In contrast. the RSVP-TE is being signalled and periodically sends keep alive messages. As such, it would be able to spot this issue. Does it mean that RSVP-TE being much older than SR-TE is better? Well, not really…<\/p>\n\n\n\n
There are multiple solutions, how you can solve problem:<\/p>\n\n\n\n
  1. You can deploy SDN controller, where you can signal the topology of the network using the BGP-LS and use the PCEP to signal tunnels. The solution is viable, but is very complex and requires a lot of investment (time or money).<\/li>
  2. You can add the possibility to SR-TE tunnels to send the end-to-end health checks.<\/li><\/ol>\n\n\n\n
    Back in the days, it was not really possible. However, now we can achieve that using the Seamless BFD (S-BFD). In a nutshell, S-BFD is a modified version of the BFD to address the scalability issue. In Nokia SR OS, it requires way more configuration than a regular BFD, but this is the only option to make the SR-TE adaptive to the link failures. The good thing is that you can tune performance to notice failures very quickly (matter of milliseconds) so that you can do a failure quickly. <\/p>\n\n\n\n
    Network setup<\/h2>\n\n\n\n
    Let’s take a look in the sample lab setup we used to test the solution:<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    We don’t show in this article the basic configuration of ISIS or Segment Routing as it is expected you already know how it works. Instead, we are focusing only on the SR-TE and seamless BFD. The IP addressing is relatively straightforward as well as schema of the SR SID’s and BFD discriminators used in the seamless BFD (see below for further details).<\/p>\n\n\n\n
    Refer to the corresponding article to learn how to configure ISIS<\/a> or Segment Routing<\/a>.<\/p><\/blockquote>\n\n\n\n
    Usage<\/h2>\n\n\n\n
    This part covers the details of the configuration of the overall solution. The full configuration is split into 2 big parts, which is the configuration of the seamless BFD framework and the SR-TE policies on top.<\/p>\n\n\n\n
    #1. Configuration of Seamless BFD<\/h3>\n\n\n\n
    The first step is to configure the Seamless BFD. Generally, its configuration is covered in the corresponding configuration guide.<\/a> However, the level of details is a little bit low, so we decided to go deeper.<\/p>\n\n\n\n
    From the configuration perspective, the seamless BFD consist of the two elements:<\/p>\n\n\n\n
    1. Reflector part (configured on the tail-end router for the SR-TE tunnel).<\/li>
    2. Peer part (configured on the head-end router). <\/li><\/ol>\n\n\n\n
      #1.1. Reflector configuration<\/h4>\n\n\n\n
      In contrast to an “ordinary” BFD, where the discriminator is automatically allocated for each BGP session, for the seamless BFD it shall be configured manually. The following configuration is implemented on SR1:<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      <\/div><\/td>
      A:SR1# configure {
      \n        bfd {
      \n            seamless-bfd {
      \n                reflector "SR1" {
      \n                    admin-state enable
      \n                    discriminator 524289
      \n                    local-state up
      \n                }
      \n            }
      \n        }
      \n    }<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      Each host shall have its own discriminator, so on the other side of the SR-TE, the SR-TE shall have a different number:<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      <\/div><\/td>
      A:SR4# configure {
      \n        bfd {
      \n          seamless-bfd {
      \n              reflector "SR4" {
      \n                  admin-state enable
      \n                  discriminator 524489
      \n                  local-state up
      \n              }
      \n          }
      \n      }
      \n  }<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      #1.2. Seamless BFD peer configuration<\/h4>\n\n\n\n
      The second part of the seamless BFD configuration is the peer: for SR4 the peer is SR1 and for SR1 the peer is SR4 in our topology. The seamless BFD session is destined to the system interface, so pay attention to the topology above for the details of the IP\/discriminator mapping. E.g., from SR1 you need to configure the IP address of the SR4 system’s interface and SR4 discriminator as seamless BFD peer.<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      <\/div><\/td>
      A:SR1# configure {
      \n          router "Base" {
      \n              bfd {
      \n                  seamless-bfd {
      \n                      peer 10.0.255.44 {
      \n                          discriminator 524489
      \n                      }
      \n                  }
      \n              }
      \n          }
      \n      }<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      And on the other side:<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      <\/div><\/td>
      A:SR4# configure {
      \n          router "Base" {
      \n              bfd {
      \n                  seamless-bfd {
      \n                      peer 10.0.255.11 {
      \n                          discriminator 524289
      \n                      }
      \n                  }
      \n              }
      \n          }
      \n      }<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      At this stage, we have created the configuration of the peers. However, they won’t be operational before we map it to the SR-TE policies.<\/p>\n\n\n\n
      #2. Configuration of SR-TE policies with Seamless BFD<\/h3>\n\n\n\n
      The mapping of the seamless BFD session to the SR-TE tunnel consist of two steps: creating a BFD policy and applying it to the SR-TE LSP.<\/p>\n\n\n\n
      #2.1. Configuration of the BFD policy<\/h4>\n\n\n\n
      Generally, this configuration shall be applied on the head-end router. However, in our cases the SR-TE tunnels are bidirectional (well, two unidirectional). Therefore, the same configuration shall be created and applied both for SR1 and SR4 ends:<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      <\/div><\/td>
      A:SR4# configure {
      \n          router "Base" {
      \n              bfd {
      \n                  bfd-template "SR_TE_STRICT_TUNNEL" {
      \n                      echo-receive 100
      \n                      multiplier 3
      \n                      receive-interval 50
      \n                      transmit-interval 50
      \n                      type cpm-np
      \n                  }
      \n              }
      \n          }
      \n      }<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      #2.2. Mapping the BFD policy to SR-TE tunnel<\/h4>\n\n\n\n
      Final configuration step is to map the created policy to the SR-TE LSP. Here is the configuration of the two SR-TE tunnels with the explicit paths and enabled BFD policies from SR1 to SR4 (configuration from SR4 to SR1 will be mirrored):<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      52
      53
      54
      55
      56
      57
      58
      59
      60
      61
      62
      63
      64
      65
      66
      67
      <\/div><\/td>
      &#91;gl:configure router "Base" mpls]
      \nA:SR1# info
      \n    admin-state enable
      \n    path "SR4_explicit_1" {
      \n        admin-state enable
      \n        hop 1 {
      \n            ip-address 10.0.0.1
      \n            type strict
      \n        }
      \n        hop 2 {
      \n            ip-address 10.0.0.5
      \n            type strict
      \n        }
      \n        hop 3 {
      \n            ip-address 10.0.0.9
      \n            type strict
      \n        }
      \n    }
      \n    path "SR4_explicit_2" {
      \n        admin-state enable
      \n        hop 1 {
      \n            ip-address 10.0.0.3
      \n            type strict
      \n        }
      \n        hop 2 {
      \n            ip-address 10.0.0.7
      \n            type strict
      \n        }
      \n        hop 3 {
      \n            ip-address 10.0.0.11
      \n            type strict
      \n        }
      \n    }
      \n    lsp "SR4_strict_1" {
      \n        admin-state enable
      \n        type p2p-sr-te
      \n        to 10.0.255.44
      \n        metric 25
      \n        path-computation-method local-cspf
      \n        max-sr-labels {
      \n            additional-frr-labels 4
      \n        }
      \n        bfd {
      \n            bfd-liveness true
      \n            bfd-template "SR_TE_STRICT_TUNNEL"
      \n            failure-action failover-or-down
      \n        }
      \n        primary "SR4_explicit_1" {
      \n        }
      \n    }
      \n    lsp "SR4_explicit_2" {
      \n        admin-state enable
      \n        type p2p-sr-te
      \n        to 10.0.255.44
      \n        metric 25
      \n        path-computation-method local-cspf
      \n        max-sr-labels {
      \n            additional-frr-labels 4
      \n        }
      \n        bfd {
      \n            bfd-liveness true
      \n            bfd-template "SR_TE_STRICT_TUNNEL"
      \n            failure-action failover-or-down
      \n        }
      \n        primary "SR4_explicit_2" {
      \n        }
      \n    }<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      The overall logic here is that you have two independent explicit paths, which are converted into the label stack using the adjacent labels. Then each LSP is mapped to a single explicit path and have the BFD template enabled. The BFD template should have an action failure-or-down, which means that the LSP will be brought down in case that BFD packets sent inside the MPLS tunnel are not returned as plain IP traffic.<\/p>\n\n\n\n
      #2.3. Fallback scenario<\/h4>\n\n\n\n
      If you carefully take a look on the topology and think about the failure scenarios, you can figure out that there are multiple cases where two outages may result in full traffic loss. For such cases we implemented the fallback to the SR-ISIS, where no load sharing is available, but connectivity continues working.<\/p>\n\n\n\n
      #3. Validation<\/h3>\n\n\n\n
      The validation of such service consist from the control plane and data plane verifications.<\/p>\n\n\n\n
      #3.1 Control plane<\/h4>\n\n\n\n
      You can check the the SR-TE LSP as follows:<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      50
      51
      <\/div><\/td>
      A:SR4# show router mpls sr-te-lsp "SR1_strict_1" detail 
      \n
      \n===============================================================================
      \nMPLS SR-TE LSPs (Originating) (Detail)
      \n===============================================================================
      \nLegend : 
      \n    + - Inherited
      \n===============================================================================
      \n-------------------------------------------------------------------------------
      \nType : Originating 
      \n-------------------------------------------------------------------------------
      \nLSP Name   : SR1_strict_1
      \nLSP Type        : SrTeLsp                   LSP Tunnel ID        : 8
      \nLSP Index       : 65543                     TTM Tunnel Id        : 655369
      \nFrom            : 10.0.255.11
      \nTo              : 10.0.255.11
      \nAdm State       : Up                        Oper State           : Up
      \nLSP Up Time     : 14d 16:01:59              LSP Down Time        : 0d 00:00:00
      \nTransitions     : 33                        Path Changes         : 33
      \nRetry Limit     : 0                         Retry Timer          : 30 sec
      \nHop Limit       : 255                       Negotiated MTU       : 9178
      \nPathCompMethod  : none                      
      \nMetric          : 25                        
      \nLocal Sr Protec*: preferred                 Label Stack Reduction: Disabled
      \nLoad Bal Wt     : N\/A                       ClassForwarding      : Disabled
      \nInclude Grps    :                           Exclude Grps         : 
      \nNone                                           None
      \nEgress Stats    : Disabled                  
      \nBFD Template    : SR_TE_STRICT_TUNNEL       BFD Ping Intvl       : N\/A
      \nBFD Enable      : True                      BFD Failure-action   : FailoverOrDn
      \nWaitForUpTimer  : 4                         
      \n 
      \nRevert Timer    : Disabled                  Next Revert In       : N\/A
      \nEntropy Label   : Enabled+                  Oper Entropy Label   : Enabled
      \nNegotiated EL   : Disabled                  
      \nVprnAutoBind    : Enabled                   
      \nIGP Shortcut    : Enabled                   BGP Shortcut         : Enabled
      \nIGP LFA         : Disabled                  IGP Rel Metric       : Disabled
      \nBGPTransTun     : Enabled                   
      \nOper Metric     : 25                        
      \nPCE Report      : Disabled+                 
      \nPCE Control     : Disabled                  
      \nMax SR Labels   : 6                         Additional FRR Labels: 4
      \nPath Profile    : None                      
      \nAdmin Tags      : None                      
      \n 
      \nPrimary(a)      : SR1_explicit_1
      \n                                            Up Time              : 14d 16:01:59
      \nBandwidth       : 0 Mbps                    
      \n===============================================================================
      \n* indicates that the corresponding row element may have been truncated.<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      You see that the BFD is enabled and up. To check the status of the seamless BFD session use:<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      <\/div><\/td>
      A:SR4# show router bfd seamless-bfd session lsp-name "SR1_strict_1" detail 
      \n
      \n===============================================================================
      \nBFD Session
      \n===============================================================================
      \nPrefix         : 10.0.255.11\/32
      \nLocal Address  : 10.0.255.44
      \nLSP Name       : SR1_strict_1
      \nLSP Index      : 65543                    Path LSP ID      : 54274
      \nFec Type       : srTe                     
      \nOper State     : Up                       Protocols        : mplsLsp 
      \nUp Time        : 14d 16:03:56             Up Transitions   : 1
      \nDown Time      : None                     Down Transitions : 0
      \n                                          Version Mismatch : 0
      \n 
      \nForwarding Information
      \n 
      \nLocal Discr    : 28                       Local State      : Up
      \nLocal Diag     : 0 (None)
      \nLocal Mode     : Demand                   
      \nLocal Min Tx   : 50                       Local Mult       : 3
      \nLast Sent (ms) : 11                       Local Min Rx     : 50
      \nType           : cpm-np                   
      \nRemote Discr   : 524289                   Remote State     : Up
      \nRemote Diag    : 0 (None)                 Remote Mode      : Async
      \nRemote Min Tx  : 50                       Remote Mult      : 3
      \nRemote C-flag  : 1                        
      \nLast Recv (ms) : 3                        Remote Min Rx    : 3
      \n===============================================================================
      \n===============================================================================<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      You can also validate the full list of the active seamless BFD sessions:<\/p>\n\n\n\n
      \n\n
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      <\/div><\/td>
      A:SR1# show router bfd seamless-bfd session lsp-path
      \n
      \n===============================================================================
      \nLegend:
      \n  Session Id = Interface Name | LSP Name | Prefix | RSVP Sess Name | Service Id
      \n  wp = Working path   pp = Protecting path
      \n===============================================================================
      \nBFD Session
      \n===============================================================================
      \nSession Id                                        State      Tx Pkts    Rx Pkts
      \n  Rem Addr\/Info\/SdpId:VcId                      Multipl     Tx Intvl   Rx Intvl
      \n  Protocols                                        Type     LAG Port     LAG ID
      \n  Loc Addr
      \n-------------------------------------------------------------------------------
      \n10.0.255.44\/32                                       Up          N\/A        N\/A
      \n  10.0.255.44                                         3           50         50
      \n  mplsLsp                                        cpm-np          N\/A        N\/A
      \n  10.0.255.11
      \n10.0.255.44\/32                                       Up          N\/A        N\/A
      \n  10.0.255.44                                         3           50         50
      \n  mplsLsp                                        cpm-np          N\/A        N\/A
      \n  10.0.255.11
      \n-------------------------------------------------------------------------------
      \nNo. of BFD sessions: 2
      \n===============================================================================<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n\n<\/pre>\n\n\n\n
      #3.2 Data plane<\/h4>\n\n\n\n
      The validation of the date plane is pretty straightforward. As long as you shut down one of the links on one path, the traffic is switched to a second LSP and vice versa within 150 ms, so customer’s traffic is slightly affected.<\/p>\n\n\n\n
      Cooperation<\/h2>\n\n\n\n
      This solution was created with the cooperation with Pau Nadeu Rabat<\/a> and implemented in the Service Provider Network in Spain. Thanks a lot for an opportunity to work together on that.<\/p>\n\n\n\n
      Lessons learned<\/h2>\n\n\n\n
      Back in the days (2019) I was already testing the SR-TE tunnels in the Vodafone. However, that days the seamless BFD was doing just its first steps and it was not yet supported in the production images of the network operation systems. It was quite an interesting exercise to brush up that experience and learn how the pure service provider technologies has evolved since that time.<\/p>\n\n\n\n
      Conclusion<\/h2>\n\n\n\n
      The Segment Routing and Segment Routing Traffic Engineering are the key building blocks of modern service provider networks. However, SR-TE lacks the mechanism of the end-to-end path validation, which existed in RSVP-TE. The seamless BFD fixes this issue, so that together they provide the high level or programmability and resiliency. Take care and good bye.<\/p>\n\n\n\n
      Support us<\/h2>\n\n\n\n