VXLAN SVI suspended as soon as VLAN becomes part of vn-segment
October 21, 2018
When configuring VXLAN with MP-BGP EVPN Control Plane I’ve come across interesting behavior. When assigning VLAN to VXLAN segment (vn-segment) my SVI for that VLAN was automatically suspended with the following error:
%FWM-3-NVE_LIBINIT: NVE library not initialized by client in nve_vni_export_get_data
|
1 2 3 4 5 6 7 8 9 10 11 |
<span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# vlan 10 N5K2(config-vlan)# vn-segment 10100 N5K2(config-vlan)# exit N5K2(config)# 2018 Oct 20 21:40:29 N5K1 %FWM-3-NVE_LIBINIT: NVE library not initialized by client in nve_vni_export_get_data N5K2(config)# sh int vlan 10 <span style="background-color: #ffff99;"><strong>Vlan10 is down (suspended), line protocol is down</strong></span></span> |
Even after following proper the proper order of operations SVI would still become suspended after enabling vn-segment.
- vlan # ; vn-segment #
- interface nve 1 ; member vni # ; mcast-group #
- interface #
So I started troubleshooting from the down up. If you are certain that all of the below are correct scroll down all the way for the remediation.
- Are all the features enabled on the Spines and Leafs?
- Is the unicast routing working (UNDERLAY) and can you reach your neighbors?
- Is the multicast routing working and can you reach igmp mcast-group members?
- Is the VNI up for the particular mcast-group and are you learning MACs via VTEPs?
- Is the control plane (MP-BGP) working and are the routes being learned?
Are all the features enabled on the Spines and Leafs?
Make sure that all the features are defined (use either OSPF or ISIS for underlay).
|
1 2 3 4 5 6 7 8 9 10 |
<span style="font-family: 'courier new', courier, monospace;">install feature-set fabric</span> <span style="font-family: 'courier new', courier, monospace;">feature-set fabric</span> <span style="font-family: 'courier new', courier, monospace;">feature fabric forwarding</span> <span style="font-family: 'courier new', courier, monospace;">feature interface-vlan</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">feature ospf</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">!feature isis</span></strong> <span style="font-family: 'courier new', courier, monospace;">feature nv overlay</span> <span style="font-family: 'courier new', courier, monospace;">feature bgp</span> <span style="font-family: 'courier new', courier, monospace;">feature vn-segment-vlan-based</span> <span style="font-family: 'courier new', courier, monospace;">nv overlay evpn</span> |
Is the unicast routing working (UNDERLAY/OSPF) and can you reach your neighbors?
If this is where it fails, verify your underlay routing. Example below from VTEP2/Leaf102 (N5K2 .52) perspective to VTEP1 (N5K1 .51) and both Spines (.71&.72).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
<span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-if)# sh ip route ospf-UNDERLAY</strong> </span> <span style="font-family: 'courier new', courier, monospace;">IP Route Table for VRF "default"</span> <span style="font-family: 'courier new', courier, monospace;">'*' denotes best ucast next-hop</span> <span style="font-family: 'courier new', courier, monospace;">'**' denotes best mcast next-hop</span> <span style="font-family: 'courier new', courier, monospace;">'[x/y]' denotes [preference/metric]</span> <span style="font-family: 'courier new', courier, monospace;">'%<string>' in via output denotes VRF <string></span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">10.0.0.51/32, ubest/mbest: 2/0</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">*via 10.0.0.71, Eth1/5, [110/9], 17:14:52, ospf-UNDERLAY, intra</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">*via 10.0.0.72, Eth1/6, [110/9], 17:14:52, ospf-UNDERLAY, intra</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">10.0.0.71/32, ubest/mbest: 1/0</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">*via 10.0.0.71, Eth1/5, [110/5], 21:23:18, ospf-UNDERLAY, intra</span></strong> <span style="font-family: 'courier new', courier, monospace;">via 10.0.0.71, Eth1/5, [250/0], 21:23:26, am</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">10.0.0.72/32, ubest/mbest: 1/0</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">*via 10.0.0.72, Eth1/6, [110/5], 21:23:20, ospf-UNDERLAY, intra</span></strong> <span style="font-family: 'courier new', courier, monospace;">via 10.0.0.72, Eth1/6, [250/0], 21:23:26, am</span> <span style="font-family: 'courier new', courier, monospace;">10.0.0.254/32, ubest/mbest: 2/0</span> <span style="font-family: 'courier new', courier, monospace;">*via 10.0.0.71, Eth1/5, [110/5], 21:23:18, ospf-UNDERLAY, intra</span> <span style="font-family: 'courier new', courier, monospace;">*via 10.0.0.72, Eth1/6, [110/5], 21:23:18, ospf-UNDERLAY, intra</span> <span style="font-family: 'courier new', courier, monospace;">10.1.2.51/32, ubest/mbest: 2/0</span> <span style="font-family: 'courier new', courier, monospace;">*via 10.0.0.71, Eth1/5, [110/9], 17:14:52, ospf-UNDERLAY, intra</span> <span style="font-family: 'courier new', courier, monospace;">*via 10.0.0.72, Eth1/6, [110/9], 17:14:52, ospf-UNDERLAY, intra</span> <span style="font-family: 'courier new', courier, monospace;">10.1.2.71/32, ubest/mbest: 1/0</span> <span style="font-family: 'courier new', courier, monospace;">*via 10.0.0.71, Eth1/5, [110/5], 17:18:20, ospf-UNDERLAY, intra</span> <span style="font-family: 'courier new', courier, monospace;">10.1.2.72/32, ubest/mbest: 1/0</span> <span style="font-family: 'courier new', courier, monospace;">*via 10.0.0.72, Eth1/6, [110/5], 17:18:20, ospf-UNDERLAY, intra </span> <strong><span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# ping 10.0.0.51</span></strong> <span style="font-family: 'courier new', courier, monospace;">PING 10.0.0.51 (10.0.0.51): 56 data bytes</span> <span style="font-family: 'courier new', courier, monospace;">64 bytes from 10.0.0.51: icmp_seq=0 ttl=253 time=1.129 ms</span> <strong><span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# ping 10.0.0.71</span></strong> <span style="font-family: 'courier new', courier, monospace;">PING 10.0.0.71 (10.0.0.71): 56 data bytes</span> <span style="font-family: 'courier new', courier, monospace;">64 bytes from 10.0.0.71: icmp_seq=0 ttl=254 time=1.076 ms</span> <strong><span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# ping 10.0.0.72</span></strong> <span style="font-family: 'courier new', courier, monospace;">PING 10.0.0.72 (10.0.0.72): 56 data bytes</span> <span style="font-family: 'courier new', courier, monospace;">64 bytes from 10.0.0.72: icmp_seq=0 ttl=254 time=1.169 ms</span> |
Is the multicast routing working and can you reach igmp mcast-group members?
If this is where it fails verify you have RP defined on the Spine (static, auto-rp or anycast) and that your VTEPs have the RP assigned. Example below from elected RP (N7K2) and VTEP2 (N5K2)
|
1 2 3 4 5 6 7 |
<strong><span style="font-family: 'courier new', courier, monospace;">N7K2(config-if-range)# sh run pim</span></strong> <span style="font-family: 'courier new', courier, monospace;">feature pim</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">ip pim send-rp-announce loopback0 group-list 224.0.0.0/4 bidir</span></strong> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">ip pim send-rp-discovery loopback0</span></strong> <span style="font-family: 'courier new', courier, monospace;">ip pim ssm range 232.0.0.0/8</span> <span style="font-family: 'courier new', courier, monospace;">ip pim bfd</span> |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
<span style="font-family: 'courier new', courier, monospace;"> <strong>N5K2(config-if)# sh ip pim rp</strong></span> <span style="font-family: 'courier new', courier, monospace;">PIM RP Status Information for VRF "default"</span> <span style="font-family: 'courier new', courier, monospace;">BSR disabled</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">Auto-RP listen-only mode</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">Auto-RP RPA: 10.0.0.72, uptime: 21:26:47, expires: 00:02:55</span></strong> <span style="font-family: 'courier new', courier, monospace;">BSR RP Candidate policy: None</span> <span style="font-family: 'courier new', courier, monospace;">BSR RP policy: None</span> <span style="font-family: 'courier new', courier, monospace;">Auto-RP Announce policy: None</span> <span style="font-family: 'courier new', courier, monospace;">Auto-RP Discovery policy: None</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">RP: 10.0.0.72, (1), </span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">uptime: 21:26:47 priority: 0, </span> <span style="font-family: 'courier new', courier, monospace;"><span style="background-color: #ffff99;">RP-source: 10.0.0.72 (A),</span> </span></strong> <span style="font-family: 'courier new', courier, monospace;">group ranges:</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">224.0.0.0/4 (bidir) , expires: 00:02:55 (A)</span></strong> |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
<span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-if)# sh ip mroute</strong> </span> <span style="font-family: 'courier new', courier, monospace;">IP Multicast Routing Table for VRF "default"</span> <span style="font-family: 'courier new', courier, monospace;">(*, 224.0.0.0/4), bidir, uptime: 21:26:50, pim ip </span> <span style="font-family: 'courier new', courier, monospace;">Incoming interface: Ethernet1/6, RPF nbr: 10.0.0.72</span> <span style="font-family: 'courier new', courier, monospace;">Outgoing interface list: (count: 1)</span> <span style="font-family: 'courier new', courier, monospace;">Ethernet1/6, uptime: 21:26:50, pim, (RPF)</span> <span style="font-family: 'courier new', courier, monospace;">(*, 231.1.1.1/32), bidir, uptime: 21:26:46, igmp ip pim </span> <span style="font-family: 'courier new', courier, monospace;">Incoming interface: Ethernet1/6, RPF nbr: 10.0.0.72</span> <span style="font-family: 'courier new', courier, monospace;">Outgoing interface list: (count: 2)</span> <span style="font-family: 'courier new', courier, monospace;">Ethernet1/6, uptime: 21:26:46, pim, (RPF)</span> <span style="font-family: 'courier new', courier, monospace;">loopback0, uptime: 21:26:46, igmp</span> <span style="font-family: 'courier new', courier, monospace;">(*, 232.0.0.0/8), uptime: 21:26:50, pim ip </span> <span style="font-family: 'courier new', courier, monospace;">Incoming interface: Null, RPF nbr: 0.0.0.0</span> <span style="font-family: 'courier new', courier, monospace;">Outgoing interface list: (count: 0)</span> <span style="background-color: #ffff99;"><strong><span style="font-family: 'courier new', courier, monospace;">(*, 239.1.1.0/32), bidir, uptime: 21:26:50, ip pim nve </span> <span style="font-family: 'courier new', courier, monospace;">Incoming interface: Ethernet1/6, RPF nbr: 10.0.0.72</span></strong></span> <span style="font-family: 'courier new', courier, monospace;">Outgoing interface list: (count: 2)</span> <span style="font-family: 'courier new', courier, monospace;">nve1, uptime: 17:40:46, nve</span> <span style="font-family: 'courier new', courier, monospace;">Ethernet1/6, uptime: 21:26:50, pim, (RPF)</span> <span style="font-family: 'courier new', courier, monospace;">--- 231.1.1.1 ping multicast statistics ---</span> <span style="font-family: 'courier new', courier, monospace;">5 packets transmitted,</span> <span style="font-family: 'courier new', courier, monospace;">0 packets received, 100% packet loss</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">N5K2(config-if)# ping multicast 231.1.1.1 interface e1/6</span></strong> <span style="font-family: 'courier new', courier, monospace;">PING 231.1.1.1 (231.1.1.1): 56 data bytes</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">64 bytes from 10.0.0.51: icmp_seq=0 ttl=253 time=1.108 ms</span></strong> <span style="font-family: 'courier new', courier, monospace;">--- 231.1.1.1 ping multicast statistics ---</span> <span style="font-family: 'courier new', courier, monospace;">5 packets transmitted,</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">From member 10.0.0.51: 5 packets received, 0.00% packet loss</span></strong> <span style="font-family: 'courier new', courier, monospace;">--- in total, 1 group member responded ---</span> |
Is the VNI up for the particular mcast-group and are you learning MACs via VTEPs?
If this is where it fails you may have misconfigured nve interface or you don’t have correct VXLAN mapping (vn-segment). NVE for specific VNI’s should be up either L2VNI or L3VNI. Example below from VTEP2
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
<span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-if)# sh run vlan 10</strong></span> <span style="font-family: 'courier new', courier, monospace;">vlan 10</span> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;"><strong>vn-segment 10100</strong></span> <span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-if)# sh run int nve 1</strong></span> <span style="font-family: 'courier new', courier, monospace;">interface nve1</span> <span style="font-family: 'courier new', courier, monospace;">no shutdown</span> <span style="font-family: 'courier new', courier, monospace;">source-interface loopback1</span> <span style="font-family: 'courier new', courier, monospace;">host-reachability protocol bgp</span> <span style="font-family: 'courier new', courier, monospace;">member vni 5000 associate-vrf</span> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;"><strong>member vni 10100</strong></span> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;"><strong>suppress-arp</strong></span> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;"><strong>mcast-group 239.1.1.0</strong></span> <span style="font-family: 'courier new', courier, monospace;">member vni 10200</span> <span style="font-family: 'courier new', courier, monospace;">suppress-arp</span> <span style="font-family: 'courier new', courier, monospace;">mcast-group 239.1.1.0</span> <span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-router)# sh nve vni</strong></span> <span style="font-family: 'courier new', courier, monospace;">Codes: CP - Control Plane DP - Data Plane </span> <span style="font-family: 'courier new', courier, monospace;">UC - Unconfigured SA - Suppress ARP </span> <span style="font-family: 'courier new', courier, monospace;">SU - Suppress Unknown Unicast</span> <span style="font-family: 'courier new', courier, monospace;">Interface VNI Multicast-group State Mode Type [BD/VRF] Flags</span> <span style="font-family: 'courier new', courier, monospace;">--------- -------- ----------------- ----- ---- ------------------ -----</span> <span style="font-family: 'courier new', courier, monospace;">nve1 5000 n/a Up CP L3 [transit-l3-vni] </span> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;"><strong>nve1 10100 239.1.1.0 Up CP L2 [10] SA</strong> </span> <span style="font-family: 'courier new', courier, monospace;">nve1 10200 239.1.1.0 Up CP L2 [20] SA</span> |
|
1 2 3 4 5 6 7 8 9 10 11 |
<span style="font-family: 'courier new', courier, monospace;"> <strong>N5K2(config-if)# sh mac address-table vlan 10</strong> Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 10 001b.2188.8075 dynamic 50 F F Eth1/2 <span style="background-color: #ffff99;"><strong>* 10 001b.218d.3d98 dynamic 0 F F nve1/10.1.2.51</strong></span> * 10 2020.0000.00aa static 0 F F sup-eth2 </span> |
Is the control plane (MP-BGP) working and are the routes being learned?
If your BGP is not up and you are not learning VTEPS addresses (Prefixes) you need verify your configuration for BGP EVPN.
|
1 2 3 4 5 6 7 8 9 10 11 |
<span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-if)# sh bgp l2vpn evpn summary</strong> </span> <span style="font-family: 'courier new', courier, monospace;">BGP summary information for VRF default, address family L2VPN EVPN</span> <span style="font-family: 'courier new', courier, monospace;">BGP router identifier 10.0.0.52, local AS number 65535</span> <span style="font-family: 'courier new', courier, monospace;">BGP table version is 168, L2VPN EVPN config peers 2, capable peers 2</span> <span style="font-family: 'courier new', courier, monospace;">7 network entries and 9 paths using 928 bytes of memory</span> <span style="font-family: 'courier new', courier, monospace;">BGP attribute entries [7/1008], BGP AS path entries [0/0]</span> <span style="font-family: 'courier new', courier, monospace;">BGP community entries [0/0], BGP clusterlist entries [2/8]</span> <span style="font-family: 'courier new', courier, monospace;">Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/<span style="background-color: #ccffff;"><strong>PfxRcd</strong></span></span> <span style="font-family: 'courier new', courier, monospace;"><span style="background-color: #ffff99;"><strong>10.1.2.71 4 65535 1121 1118 168 0 0 18:21:22</strong></span> <strong><span style="background-color: #ccffff;">2</span> </strong></span> <span style="font-family: 'courier new', courier, monospace;"><span style="background-color: #ffff99;"><strong>10.1.2.72 4 65535 1126 1122 168 0 0 18:21:12</strong></span> <span style="background-color: #ccffff;"><strong>2</strong></span></span> |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 |
<span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-if)# sh bgp l2vpn evpn</strong> </span> <span style="font-family: 'courier new', courier, monospace;">BGP routing table information for VRF default, address family L2VPN EVPN</span> <span style="font-family: 'courier new', courier, monospace;">BGP table version is 168, local router ID is 10.0.0.52</span> <span style="font-family: 'courier new', courier, monospace;">Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best</span> <span style="font-family: 'courier new', courier, monospace;">Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected</span> <span style="font-family: 'courier new', courier, monospace;">Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup</span> <span style="font-family: 'courier new', courier, monospace;"> Network Next Hop Metric LocPrf Weight Path</span> <span style="font-family: 'courier new', courier, monospace;">Route Distinguisher: 10.0.0.51:32777</span> <span style="font-family: 'courier new', courier, monospace;">* i[2]:[0]:[0]:[48]:[001b.218d.3d98]:[0]:[0.0.0.0]/216</span> <span style="font-family: 'courier new', courier, monospace;"> 10.1.2.51 100 0 i</span> <span style="font-family: 'courier new', courier, monospace;">*>i 10.1.2.51 100 0 i</span> <span style="font-family: 'courier new', courier, monospace;">*>i[2]:[0]:[0]:[48]:[001b.218d.3d98]:[32]:[10.0.1.11]/272</span> <span style="font-family: 'courier new', courier, monospace;"> 10.1.2.51 100 0 i</span> <span style="font-family: 'courier new', courier, monospace;">* i 10.1.2.51 100 0 i</span> <span style="font-family: 'courier new', courier, monospace;">Route Distinguisher: 10.0.0.52:32777 (L2VNI 10100)</span> <span style="font-family: 'courier new', courier, monospace;">*>l[2]:[0]:[0]:[48]:[001b.2188.8075]:[0]:[0.0.0.0]/216</span> <span style="font-family: 'courier new', courier, monospace;"> 10.1.2.52 100 32768 i</span> <span style="font-family: 'courier new', courier, monospace;">*>i[2]:[0]:[0]:[48]:[001b.218d.3d98]:[0]:[0.0.0.0]/216</span> <span style="font-family: 'courier new', courier, monospace;"> 10.1.2.51 100 0 i</span> <span style="font-family: 'courier new', courier, monospace;">*>l[2]:[0]:[0]:[48]:[001b.2188.8075]:[32]:[10.0.1.12]/272</span> <span style="font-family: 'courier new', courier, monospace;"> 10.1.2.52 100 32768 i</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">*>i[2]:[0]:[0]:[48]:[001b.218d.3d98]:[32]:[10.0.1.11]/272</span></strong> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;"> 10.1.2.51 100 0 i</span></strong> <strong><span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# sh bgp l2vpn evpn 10.0.1.11</span></strong> <span style="font-family: 'courier new', courier, monospace;">Route Distinguisher: 10.0.0.51:32777</span> <span style="font-family: 'courier new', courier, monospace;">BGP routing table entry for [2]:[0]:[0]:[48]:[001b.218d.3d98]:[32]:[10.0.1.11]/272, version 166</span> <span style="font-family: 'courier new', courier, monospace;">Paths: (2 available, best #1)</span> <span style="font-family: 'courier new', courier, monospace;">Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is not in HW, , is locked</span> <span style="font-family: 'courier new', courier, monospace;"> Advertised path-id 1</span> <span style="background-color: #ffff99;"><strong><span style="font-family: 'courier new', courier, monospace;"> Path type: internal, path is valid, is best path</span></strong></span> <span style="background-color: #ffff99;"><strong><span style="font-family: 'courier new', courier, monospace;"> AS-Path: NONE, path sourced internal to AS</span></strong></span> <span style="background-color: #ffff99;"><strong><span style="font-family: 'courier new', courier, monospace;"> 10.1.2.51 (metric 9) from 10.1.2.71 (10.1.2.71)</span></strong></span> <span style="font-family: 'courier new', courier, monospace;"> Origin IGP, MED not set, localpref 100, weight 0</span> <span style="font-family: 'courier new', courier, monospace;"> Received label 10100 5000</span> <span style="font-family: 'courier new', courier, monospace;"> Extcommunity: </span> <span style="font-family: 'courier new', courier, monospace;"> RT:65535:5000</span> <span style="font-family: 'courier new', courier, monospace;"> RT:65535:10100</span> <span style="font-family: 'courier new', courier, monospace;"> ENCAP:8</span> <span style="font-family: 'courier new', courier, monospace;"> Router MAC:00de.fb12.1a7c</span> <span style="font-family: 'courier new', courier, monospace;"> Originator: 10.0.0.51 Cluster list: 10.1.2.71 </span> <span style="font-family: 'courier new', courier, monospace;"> Path type: internal, path is valid, not best reason: Neighbor Address</span> <span style="font-family: 'courier new', courier, monospace;"> AS-Path: NONE, path sourced internal to AS</span> <span style="font-family: 'courier new', courier, monospace;"> 10.1.2.51 (metric 9) from 10.1.2.72 (10.1.2.72)</span> <span style="font-family: 'courier new', courier, monospace;"> Origin IGP, MED not set, localpref 100, weight 0</span> <span style="font-family: 'courier new', courier, monospace;"> Received label 10100 5000</span> <span style="font-family: 'courier new', courier, monospace;"> Extcommunity: </span> <span style="font-family: 'courier new', courier, monospace;"> RT:65535:5000</span> <span style="font-family: 'courier new', courier, monospace;"> RT:65535:10100</span> <span style="font-family: 'courier new', courier, monospace;"> ENCAP:8</span> <span style="font-family: 'courier new', courier, monospace;"> Router MAC:00de.fb12.1a7c</span> <span style="font-family: 'courier new', courier, monospace;"> Originator: 10.0.0.51 Cluster list: 10.1.2.72 </span> <span style="font-family: 'courier new', courier, monospace;"> Path-id 1 not advertised to any peer</span> <span style="background-color: #ffff99;"><strong><span style="font-family: 'courier new', courier, monospace;">Route Distinguisher: 10.0.0.52:32777 (L2VNI 10100)</span></strong></span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">BGP routing table entry for [2]:[0]:[0]:[48]:[001b.218d.3d98]:[32]:[10.0.1.11]/272, version 165</span> <span style="font-family: 'courier new', courier, monospace;">Paths: (1 available, best #1)</span> <span style="font-family: 'courier new', courier, monospace;">Flags: (0x00021a) on xmit-list, is in l2rib/evpn, is not in HW, </span> <span style="font-family: 'courier new', courier, monospace;"> Advertised path-id 1</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;"> Path type: internal, path is valid, is best path</span> <span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;"> Imported from 10.0.0.51:32777:[2]:[0]:[0]:[48]:[001b.218d.3d98]:[32]:[10.0.1.11]/272 </span> <span style="font-family: 'courier new', courier, monospace;"> AS-Path: NONE, path sourced internal to AS</span> <span style="font-family: 'courier new', courier, monospace;"> 10.1.2.51 (metric 9) from 10.1.2.71 (10.1.2.71)</span> <span style="font-family: 'courier new', courier, monospace;"> Origin IGP, MED not set, localpref 100, weight 0</span> <span style="font-family: 'courier new', courier, monospace;"> Received label 10100 5000</span> <span style="font-family: 'courier new', courier, monospace;"> Extcommunity: </span> <span style="font-family: 'courier new', courier, monospace;"> RT:65535:5000</span> <span style="font-family: 'courier new', courier, monospace;"> RT:65535:10100</span> <span style="font-family: 'courier new', courier, monospace;"> ENCAP:8</span> <span style="font-family: 'courier new', courier, monospace;"> Router MAC:00de.fb12.1a7c</span> <span style="font-family: 'courier new', courier, monospace;"> Originator: 10.0.0.51 Cluster list: 10.1.2.71 </span> <span style="font-family: 'courier new', courier, monospace;"> Path-id 1 not advertised to any peer</span> |
If all from the above checks out please consider this:
- Per Cisco support you should be using separate loopback address for VTEP/Unicast Underlay/BGP. Otherwise this is no supported configuration. That being said try to reconfigure your BGP peering with different loopback address. Even though I haven’t seen issues with using i.e lo0 for all.
- If the anycast gateway feature is enabled for a specific VNI, then the anyway gateway feature must be enabled on all VTEPs that have that VNI configured. Having the anycast gateway feature configured on only some of the VTEPs enabled for a specific VNI is not supported.
- In plain text if you are tend to use fabric forwarding anycast gateway for seamless VM mobility across your VTEPS your SVI needs to be the part of it – otherwise remove that global command In example if you want to vMotion your VM from one VTEP to another fabric forwarding anycast will make that transfer painless when underline VM last resort MAC address will not change and will be able to forward traffic without any interruptions. This is a good feature.
Reason why SVI was being suspended as soon as it becomes vn-segment was due to having fabric forwarding anycast enabled where SVI was not part of it.
|
1 2 3 4 5 6 7 8 9 10 11 |
<span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# sh run | in anycast</span> <span style="background-color: #ffff99;"><strong><span style="font-family: 'courier new', courier, monospace;">fabric forwarding anycast-gateway-mac 2020.0000.00AA</span></strong></span> <span style="font-family: 'courier new', courier, monospace;">fabric forwarding mode anycast-gateway</span> <span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# sh run int vlan 10</span> <span style="font-family: 'courier new', courier, monospace;">interface Vlan10</span> <span style="font-family: 'courier new', courier, monospace;">no shutdown</span> <span style="font-family: 'courier new', courier, monospace;">vrf member transit-l3-vni</span> <span style="font-family: 'courier new', courier, monospace;">ip address 10.0.1.52/24</span> |
As soon as I enabled fabric forwarding under the SVI the suspended tag went away.
|
1 2 3 4 5 6 7 |
<span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# sh int vlan 10 </span><strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">Vlan10 is down (suspended), line protocol is down </span></strong> <span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# int vlan 10</span> <strong><span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# <span style="background-color: #ffff99;">fabric forwarding mode anycast-gateway</span></span></strong> <span style="font-family: 'courier new', courier, monospace;">N5K2(config-if)# sh int vlan 10</span> <strong><span style="font-family: 'courier new', courier, monospace; background-color: #ffff99;">Vlan10 is up, line protocol is up</span></strong> |
I have also noticed that L3VNI kicks in right away and its added to route distinguisher.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
<span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-if)# sh bgp l2vpn evpn</strong> </span> <span style="font-family: 'courier new', courier, monospace;">BGP routing table information for VRF default, address family L2VPN EVPN</span> <span style="font-family: 'courier new', courier, monospace;">BGP table version is 170, local router ID is 10.0.0.52</span> <span style="font-family: 'courier new', courier, monospace;">Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best</span> <span style="font-family: 'courier new', courier, monospace;">Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected</span> <span style="font-family: 'courier new', courier, monospace;">Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup</span> <strong><span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;">Route Distinguisher: 10.0.0.52:3 (L3VNI 5000)</span></strong> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;">*>i[2]:[0]:[0]:[48]:[001b.218d.3d98]:[32]:[10.0.1.11]/272</span> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;">10.1.2.51 100 0 i </span> |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
<span style="font-family: 'courier new', courier, monospace;"><strong>N5K2(config-if)# sh bgp l2vpn evpn 10.0.1.11</strong></span> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;">Route Distinguisher: 10.0.0.52:3 (L3VNI 5000)</span> <span style="background-color: #ffff99; font-family: 'courier new', courier, monospace;">BGP routing table entry for [2]:[0]:[0]:[48]:[001b.218d.3d98]:[32]:[10.0.1.11]/272, version 164</span> <span style="font-family: 'courier new', courier, monospace;">Paths: (1 available, best #1)</span> <span style="font-family: 'courier new', courier, monospace;">Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is not in HW,</span> <span style="font-family: 'courier new', courier, monospace;">Advertised path-id 1</span> <span style="font-family: 'courier new', courier, monospace;">Path type: internal, path is valid, is best path</span> <span style="font-family: 'courier new', courier, monospace;"><span style="background-color: #ffff99;">Imported from 10.0.0.51:32777:[2]:[0]:[0]:[48]:[001b.218d.3d98]:[32]:[10.0.1.11]/272</span> </span> <span style="font-family: 'courier new', courier, monospace;">AS-Path: NONE, path sourced internal to AS</span> <span style="font-family: 'courier new', courier, monospace;">10.1.2.51 (metric 9) from 10.1.2.71 (10.1.2.71)</span> <span style="font-family: 'courier new', courier, monospace;">Origin IGP, MED not set, localpref 100, weight 0</span> <span style="font-family: 'courier new', courier, monospace;">Received label 10100 5000</span> <span style="font-family: 'courier new', courier, monospace;">Extcommunity: </span> <span style="font-family: 'courier new', courier, monospace;">RT:65535:5000</span> <span style="font-family: 'courier new', courier, monospace;">RT:65535:10100</span> <span style="font-family: 'courier new', courier, monospace;">ENCAP:8</span> <span style="font-family: 'courier new', courier, monospace;">Router MAC:00de.fb12.1a7c</span> <span style="font-family: 'courier new', courier, monospace;">Originator: 10.0.0.51 Cluster list: 10.1.2.71</span> <span style="font-family: 'courier new', courier, monospace;">Path-id 1 not advertised to any peer</span> |
Bottom line is if you want to use anycast for your VTEPs make sure SVIs are part of it as well. Otherwise, don’t install that global configuration. Big thanks to MM from CCIE DC Study Group for helping me nail this down.
References:
- https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_VXLAN_Configuration_Guide_7x_chapter_0100.html
- https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/200952-Configuration-and-Verification-VXLAN-wit.html
- https://www.cisco.com/c/en/us/td/docs/switches/datacenter/pf/configuration/guide/b-pf-configuration/b-pf-configuration_chapter_01111.html#concept_B54E8C5F82C14C7A9733639B1A560A01
About The Author
Bart Dworzanczyk
Bart is passionate about new technologies and their impact on our lives. He does not believe in titles or amount of certifications but positive attitude and motivation. Simply the guy that make things happen. You can reach him via Linkedin or meet him on CSGO. Currently focusing on architecting and designing custom-build hybrid cloud solutions around IaaS, DRaaS, BaaS realm.