My 2 cents about ACI l2out – its basics, caveats and considerations

When configuring my first L2out I found very minimum documentation explaining the technical side of configuring L2out, its caveats and limitations. Thus the reason for this post because something simple can become a bottleneck while there are no good troubleshooting tools available on ACI platform that I know of. Either you know the concept or you don’t where you can end up with an error like Encap Already Used in Another EPG, debug message: encap-already-in-use.

 Encap Already Used in Another EPG, debug message: encap-already-in-use:

 

Basics

L2out is just one of the ways you can extend ACI to outside network via L2 domain. Especially, when extending to legacy networks ACI was meant to be a trust boundary. Any traffic entering or exiting ACI fabric will be inspected and dropped if you don’t have filters applied.

In practicality L2out is not very popular way to extend to legacy because of its granularity and ACI to legacy VLAN encapsulation mapping.  You need to know all the traffic in order to create your rules correctly and that creates a problem. How many customers really know their applications dependencies and fully understand their traffic pattern?

Typically this will be wide open so data collection can happen on the ACI fabric to create proper whitelist later down the line (which probably won’t happen).

Going back to the basics, we can extend layer 2 domain by three ways:

  • Extend the EPG out of the ACI Fabric
  • Extend the Bridge Domain (BD) of out the ACI fabric
  • Extend the layer 2 domain with remote VTEP (future – don’t know much about it)
extend epg vs extend bridge domain
Source Cisco

Within the Bridge Domain (BD) you can have multiple subnets so i.e. Dev team doesn’t need to engage network team to spin up new networks. Multiple EPGs belonging to the same Bridge Domain can use the same subnet till they ran out of IPs. Once that happens they can create new subnet within the same BD and continue to allocate.

The only restriction is the communication between EPGs.  Inter-EPG communication is denied by default and contracts are required unless you remove enforcement from the BD VRF.

aci bridge domain l2out
Single Bridge Domain containing multiple subnets for particular EPG. Subnets can be shared across different EPGs when using same BD.

 

Question is would you design it this way or should you have separate BD for each EPG? Remember single BD is a single Broadcast Domain which is something that we have been fighting forever. ACI introduces some optimizations for ARP and BUM but still something to think about.

Considerations and Limitations

Extending EPG out of the Fabric

  • When assigning physical interface to EPG (Static Ports) only one VLAN can be carried over that link. To accomplish multiple VLANs trunking you would need to create another EPG and reuse the same port but with unique VLAN tag. Scripting would be handy in this example. Otherwise this task becomes cumbersome.
  • Since the endpoint learning, data forwarding, and policy enforcement remain the same you are bypassing vrf enforcement and no contracts are required.
  • Extending EPG belongs to domain type Physical Domain which needs to be assigned to Domain under EPG. Can be reused with other EPGs as long as its VLAN pool contains these tags
  • Some use cases: physical workload, hypervisor that doesn’t integrates with ACI, legacy network, legacy l2 last resort, L2 DCI

Extending Bridge Domain out of the Fabric (L2out)

  • When you create External Bridged Network (L2out) you create external EPG. Rules are changing since this becomes Inter-EPG communication which is not allowed and contracts will be required between the inner EPG and outside networks.
  • (Important) EPG VLAN Mapping. EPG with L2out means two EPGs (internal and external). You cannot have two EPGs using the same VLAN tag. Meaning your internal EPG VLAN tag cannot be used for encapsulation on l2out bridge because you are required to have one to one relationship or unique VLAN to VNI mapping for forwarding across the fabric. Remember ACI uses VXLAN to encap and decap to bridge and route the traffic within the fabric.

    l2out encapsulation
    Encapsulation to legacy network happens on tag 70 where EPG belongs to 170. Tag 70 is expected on legacy network ingress interface.
  • If you have brownfield deployment and you are introducing ACI and start deploying resources under it, be aware that VLAN tag within ACI cannot equal to legacy network because your L2out won’t be able to translate that due to encapsulation already in use: Encap Already Used in Another EPG, debug message: encap-already-in-use. This introduces VLAN mapping between your legacy and ACI fabric which you need to maintain and can become cumbersome. There are use cases to use this method but I’m not aware of any.
  • (Important) Extending Bridge Domain belongs to domain type Bridged Domain which needs to be assigned to Domain under EPG.

    EPG L2 External Domain
    EPG requires L2 External Domain for L2out
  • (Important) VLAN Pool used for External Bridged Domains  must be different than the encap VLAN used for the inner EPG. In other words the encapsulation that you use for the EPG that is trying to get out needs to be different than the one on legacy side.

    l2out vlan pool
    VLAN pool facing egress from ACI to Legacy. Legacy expecting these tags i.e: tag 70 here.
  • Cleaner solution to successfully extend ACI fabric to legacy would be by creating another internal EPG. Than use static binding for the external connectivity and construct contracts between your EPGs. Concept similar to DMZ/INSIDE per say but lot cleaner and easier to maintain than L2out bridged.

Verifications

VLAN to Interface bonding verification. If that is missing there is a failure in connecting object workflow. From APIC:

VLAN to VXLAN encap mapping to Interface.

Same encap mapping can be found on ACI UI under internal EPG or L2out external EPG

epg encap operational
Internal EPG Client Endpoints
epg encap operational
External EPG Client Endpoints

 

That’s about it. Thank you.

Bart

 

Reference:

 

Tags:, , ,
2 Comments

Add a Comment

Your email address will not be published. Required fields are marked *