UCS Disjoint Layer-2 Networks – Just another tag

Disjoint Layer 2 Network feature has been around since 2.0 UCSM. That being said this is not something new that you’ll find in this post (other than possibly UCSM 3.2). This is more for my own sake trying to understand why would you use this feature or who would be the candidate for this design.

The main principle of disjoint layer 2 is the physical and logical separation to the upstream. The reasoning per Cisco documentation is that by default “all data traffic in Cisco UCS works on a principle of mutual inclusion. All traffic for all VLANs and upstream networks travels along all uplink ports and port channels”.

In the simple context what this feature does is simple pruning – that’s it. You define your networks (VLANS) and assign them to specific interfaces to get the disjoint to work. Only defined networks will be able to traverse via designated ports.

Source – Cisco

Personally, I don’t think this design is very popular and let me explain why (please correct me if I’m wrong). Within Data Center realm where physical ports are expensive you want to make sure that all of the ports are active and pushing traffic. I don’t want to burn and dedicate physical ports just for couple networks. I would use all of the bandwidth available upstream and if I want to isolate the traffic between different networks I’ll use my security appliance.

Only reason why I would use disjoint layer 2 capabilities is within Service Provider realm.  If I’m in business of selling Infrastructure as a Service (IAAS) where I can literally curve up portion of my UCS to one of my tenants and let them manage that piece then I understand. Dedicate couple fabric interconnect interfaces and uplink them to dedicated tenant switches. At that point I really want to use disjoint layer 2 because I don’t want my UCS to push my internal traffic over to the other tenant.

Even though if the other side is pruning and traffic will be dropped what if someone configures SPAN on that ingress interface and copies all the data. This could be huge vulnerability.

Disjoint Layer 2 Networks can only run on End-Host Mode fabric interconnects so this is something to consider. Main benefits of the End-Host Mode are:

  • Spanning Tree Protocol is not run on both the uplink ports
  • MAC address learning occurs only on the server ports and appliance ports
  • MAC address aging is not supported; MAC address changes are fully supported
  • Active-active links are used regardless of the number of uplink switches
  • The solution is highly scalable because the control plane is not occupied
  • All uplink ports connect to the same Layer 2 cloud

I guess that’s enough of writing, lets get the hands dirty. Diagram from below (Fig. 1) represents my LAB which I’ll use to build Disjoint Layer-2. In the real deployment you would use redundant uplinks and have them channeled up but for simplicity I’m using single uplinks.

Fig. 1 – Disjointed L2 Wiring Diagram

What I’m trying to accomplish is to have VLAN 10 network be able only to traverse via both Fabric Interconnects to upstream switch N5K9. At the same time all traffic for VLAN 20 will be skewed to N5K10.

  • VLAN 10 Disjoint L2 (FIA Ethernet1/15, FIB Ethernet1/17)
  • VLAN 20 Disjoint L2 (FIA Ethernet1/17, FIB Ethernet1/15)

Once everything is configured I’ll hop on the Fabric Interconnects to verify connectivity from the CLI standpoint. Assumption is that the upstream configuration has been configured and we have up/up on our interfaces. Basically, interfaces are configured as Uplink Ports.

  1. First of all let’s verify that our Fabric Interconnects are running End-Host Mode. This can be found under Equipment Tab under Fabric Interconnects.
  2. Create our two VLANS (10,20) This can be done under LAN Tab > VLANs > Add
  3. Once the VLANS are created let’s review our configuration from the CLI standpoint i.e from FIA upstream to N5K9 and N5K10.

    From the output we can observe that both interfaces trunking both VLANs (10 and 20) which is be design. Show platform provides additional information on the members which are part of each VLAN. You could also see from the output a concept called “designated receiver”. UCS designates single interface for each network to deal with the BUM traffic.

    Good post by Brad around forwarding concept within end-host mode and designated receiver responsibilities:

    I will only pay attention to broadcasts received from my servers or received on my designated Broadcast uplink.”

    If I receive broadcast traffic from my designated Broadcast link, I will send the broadcast to my servers but not to my other uplinks.”

    If I receive broadcast traffic from my designated Broadcast link that originated from one of my own servers, I will just drop this traffic.”

  4. Next we need to navigate to LAN Uplinks Manager and start disjoint configuration. Navigate to LAN > LAN Cloud > LAN Uplinks Manager
  5. Under LAN Uplinks Manager we navigate to VLANs Tab > VLAN Manager subtab and select Fabric A or Fabric B to designate interfaces for specific Under appropriate Fabric select the interface and highlight to which VLAN should it belong to (if trunking more vlans you can select more my holding Ctrl key and adding more). Once ready to assign click on Add to VLAN/VLAN Group and Apply.To refresh our assignments:VLAN 10 Disjoint L2 (FIA Ethernet1/15, FIB Ethernet1/17)
    VLAN 20 Disjoint L2 (FIA Ethernet1/17, FIB Ethernet1/15)

    Repeat the process for all uplinks on both Fabric Interconnects
  6. Once the VLANs are assigned to appropriate interfaces the final LAN Uplinks Manager output should be matching our wiring diagram:
  7. CLI also confirms that our configuration is complete:


Now, we need to make sure that appropriate VLANS are assigned to vNICs or vNICs templates and the rest of the work needs to happen on VMware side of things. You’ll just need to tag appropriate VLAN to vmnic(if not using NIC Teaming) and there is your Disjoint Layer 2.










Add a Comment

Your email address will not be published. Required fields are marked *