My 2 cents about ACI l2out – its basics, caveats and considerations
October 25, 2018
When configuring my first L2out I found very minimum documentation explaining the technical side of configuring L2out, its caveats and limitations. Thus the reason for this post because something simple can become a bottleneck while there are no good troubleshooting tools available on ACI platform that I know of. Either you know the concept or you don’t where you can end up with an error like Encap Already Used in Another EPG, debug message: encap-already-in-use.
L2out is just one of the ways you can extend ACI to outside network via L2 domain. Especially, when extending to legacy networks ACI was meant to be a trust boundary. Any traffic entering or exiting ACI fabric will be inspected and dropped if you don’t have filters applied.
In practicality L2out is not very popular way to extend to legacy because of its granularity and ACI to legacy VLAN encapsulation mapping. You need to know all the traffic in order to create your rules correctly and that creates a problem. How many customers really know their applications dependencies and fully understand their traffic pattern?
Typically this will be wide open so data collection can happen on the ACI fabric to create proper whitelist later down the line (which probably won’t happen).
Going back to the basics, we can extend layer 2 domain by three ways:
Extend the EPG out of the ACI Fabric
Extend the Bridge Domain (BD) of out the ACI fabric
Extend the layer 2 domain with remote VTEP (future – don’t know much about it)
Within the Bridge Domain (BD) you can have multiple subnets so i.e. Dev team doesn’t need to engage network team to spin up new networks. Multiple EPGs belonging to the same Bridge Domain can use the same subnet till they ran out of IPs. Once that happens they can create new subnet within the same BD and continue to allocate.
The only restriction is the communication between EPGs. Inter-EPG communication is denied by default and contracts are required unless you remove enforcement from the BD VRF.
Question is would you design it this way or should you have separate BD for each EPG? Remember single BD is a single Broadcast Domain which is something that we have been fighting forever. ACI introduces some optimizations for ARP and BUM but still something to think about.
Considerations and Limitations
Extending EPG out of the Fabric
When assigning physical interface to EPG (Static Ports) only one VLAN can be carried over that link. To accomplish multiple VLANs trunking you would need to create another EPG and reuse the same port but with unique VLAN tag. Scripting would be handy in this example. Otherwise this task becomes cumbersome.
Since the endpoint learning, data forwarding, and policy enforcement remain the same you are bypassing vrf enforcement and no contracts are required.
Extending EPG belongs to domain type Physical Domain which needs to be assigned to Domain under EPG. Can be reused with other EPGs as long as its VLAN pool contains these tags
Some use cases: physical workload, hypervisor that doesn’t integrates with ACI, legacy network, legacy l2 last resort, L2 DCI
Extending Bridge Domain out of the Fabric (L2out)
When you create External Bridged Network (L2out) you create external EPG. Rules are changing since this becomes Inter-EPG communication which is not allowed and contracts will be required between the inner EPG and outside networks.
(Important) EPG VLAN Mapping. EPG with L2out means two EPGs (internal and external). You cannot have two EPGs using the same VLAN tag. Meaning your internal EPG VLAN tag cannot be used for encapsulation on l2out bridge because you are required to have one to one relationship or unique VLAN to VNI mapping for forwarding across the fabric. Remember ACI uses VXLAN to encap and decap to bridge and route the traffic within the fabric.
If you have brownfield deployment and you are introducing ACI and start deploying resources under it, be aware that VLAN tag within ACI cannot equal to legacy network because your L2out won’t be able to translate that due to encapsulation already in use: Encap Already Used in Another EPG, debug message: encap-already-in-use. This introduces VLAN mapping between your legacy and ACI fabric which you need to maintain and can become cumbersome. There are use cases to use this method but I’m not aware of any.
(Important) Extending Bridge Domain belongs to domain type Bridged Domain which needs to be assigned to Domain under EPG.
(Important) VLAN Pool used for External Bridged Domains must be different than the encap VLAN used for the inner EPG. In other words the encapsulation that you use for the EPG that is trying to get out needs to be different than the one on legacy side.
Cleaner solution to successfully extend ACI fabric to legacy would be by creating another internal EPG. Than use static binding for the external connectivity and construct contracts between your EPGs. Concept similar to DMZ/INSIDE per say but lot cleaner and easier to maintain than L2out bridged.
VLAN to Interface bonding verification. If that is missing there is a failure in connecting object workflow. From APIC:
apic1# show vlan detail
vlanscope: L (Portlocal). Default is global
vlan-domain : EPG170_Phys_Domain (static) Domain-Type : 'phys'
vlan : 160(static, external) 170(static, external)
Leaf Interface Vlan Type Usage Operational State Operational Vlan
------------ ---------------- ---- ----------- -------------------- ------------------------- ---------------
101 eth1/1 160 App-Epg Tenant: Rack_1 EPG_VLAN170_BD: up EPG_VLAN170_BD:
App: APP01 VLAN160: up vlan-1
Epg: VLAN160 eth1/1: formed VLAN160: vlan-14
101 eth1/1 170 App-Epg Tenant: Rack_1 EPG_VLAN170_BD: up EPG_VLAN170_BD:
App: APP01 VLAN170: up vlan-1
Epg: VLAN170 eth1/1: formed VLAN170: vlan-9
Bart is passionate about new technologies and their impact on our lives. He does not believe in titles or amount of certifications but positive attitude and motivation. Simply the guy that make things happen. You can reach him via Linkedin or meet him on CSGO. Currently focusing on architecting and designing custom-build hybrid cloud solutions around IaaS, DRaaS, BaaS realm.