VXLAN SVI suspended as soon as VLAN becomes part of vn-segment

When configuring VXLAN with MP-BGP EVPN Control Plane I’ve come across interesting behavior. When assigning VLAN to VXLAN segment (vn-segment) my SVI for that VLAN was automatically suspended with the following error:

%FWM-3-NVE_LIBINIT: NVE library not initialized by client in nve_vni_export_get_data

Even after following proper the proper order of operations SVI would still become suspended after enabling vn-segment.

  • vlan # ; vn-segment #
  • interface nve 1 ; member vni # ; mcast-group #
  • interface #

So I started troubleshooting from the down up. If you are certain that all of the below are correct scroll down all the way for the remediation.

  • Are all the features enabled on the Spines and Leafs?
  • Is the unicast routing working (UNDERLAY) and can you reach your neighbors?
  • Is the multicast routing working and can you reach igmp mcast-group members?
  • Is the VNI up for the particular mcast-group and are you learning MACs via VTEPs?
  • Is the control plane (MP-BGP) working and are the routes being learned?

Are all the features enabled on the Spines and Leafs?

Make sure that all the features are defined (use either OSPF or ISIS for underlay).

Is the unicast routing working (UNDERLAY/OSPF) and can you reach your neighbors?

If this is where it fails, verify your underlay routing. Example below from VTEP2/Leaf102 (N5K2 .52) perspective to VTEP1 (N5K1 .51) and both Spines (.71&.72).

Is the multicast routing working and can you reach igmp mcast-group members?

If this is where it fails verify you have RP defined on the Spine (static, auto-rp or anycast) and that your VTEPs have the RP assigned. Example below from elected RP (N7K2) and VTEP2 (N5K2)

Is the VNI up for the particular mcast-group and are you learning MACs via VTEPs?

If this is where it fails you may have misconfigured nve interface  or you don’t have correct VXLAN mapping (vn-segment). NVE for specific VNI’s should be up either L2VNI or L3VNI. Example below from VTEP2

Is the control plane (MP-BGP) working and are the routes being learned?

If your BGP is not up and you are not learning VTEPS addresses (Prefixes) you need verify your configuration for BGP EVPN.

If all from the above checks out please consider this:

  1. Per Cisco support you should be using separate loopback address for VTEP/Unicast Underlay/BGP. Otherwise this is no supported configuration. That being said try to reconfigure your BGP peering with different loopback address. Even though I haven’t seen issues with using i.e lo0 for all.
  2. If the anycast gateway feature is enabled for a specific VNI, then the anyway gateway feature must be enabled on all VTEPs that have that VNI configured. Having the anycast gateway feature configured on only some of the VTEPs enabled for a specific VNI is not supported.
    • In plain text if you are tend to use fabric forwarding anycast gateway for seamless VM mobility across your VTEPS your SVI needs to be the part of it – otherwise remove that global command  In example if you want to vMotion your VM from one VTEP to another fabric forwarding anycast will make that transfer painless when underline VM last resort MAC address will not change and will be able to forward traffic without any interruptions. This is a good feature.

Reason why SVI was being suspended as soon as it becomes vn-segment was due to having fabric forwarding anycast enabled where SVI was not part of it.

As soon as I enabled fabric forwarding under the SVI the suspended tag went away.

I have also noticed that L3VNI kicks in right away and its added to route distinguisher.

Bottom line is if you want to use anycast for your VTEPs make sure SVIs are part of it as well.  Otherwise, don’t install that global configuration.  Big thanks to MM from CCIE DC Study Group for helping me nail this down.

References:

Add a Comment

Your email address will not be published. Required fields are marked *