Unable to SSH to standby unit over anyconnect VPN – Drop-reason: (no-adjacency)
February 2, 2018
If you are running pair of ASA’s in active/standby you may encounter issue with connecting to your standby/secondary unit. Reason why I’m saying “may” is because this is ONLY applicable to units where anyconnect is “terminated”. To clarify Anyconnect termination I mean the edge firewalls that provide Anyconnect VPN L2L feature. Anything southbound from the termination will be reachable as long as your split-tunnel allows it in your ACL(Fig.1).
For me at first this was confusing because I was able to reach active/standby firewalls behind/southbound of my Anyconnect termination. That being said I’ve decided to dig deeper on what is the root cause.
Tcpdump capture will provide live data for analysis but before let’s find out the source from Anyconnect.
ASA/sec/stby# sh vpn-sessiondb anyconnect filter name userName
Session Type: AnyConnect
Username : userName Index : 5 Assigned IP : 10.255.4.155 Public IP : x.x.x.x Protocol : AnyConnect-Parent SSL-Tunnel DTLS-Tunnel License : AnyConnect Premium Encryption : AnyConnect-Parent: (1)none SSL-Tunnel: (1)AES128 DTLS-Tunnel: (1)AES128 Hashing : AnyConnect-Parent: (1)none SSL-Tunnel: (1)SHA1 DTLS-Tunnel: (1)SHA1 Bytes Tx : 327684 Bytes Rx : 962067 Group Policy : GRP Tunnel Group : GRP Login Time : 09:20:06 cst Fri Feb 2 2018 Duration : 0h:41m:54s Inactivity : 0h:00m:00s NAC Result : Unknown VLAN Mapping : N/A VLAN : none
Finding out the sourceof this test will help narrow down the tcpdump capture.
ASA/sec/stby(config)# cap asp-drop type asp-drop all circular-buffer match ip host 10.255.4.155 host 22.214.171.124
Once the capture is on it’s time to ssh to secondary device again. Verifying capture shows increasing buffer for just created tcpdump.
ASA/sec/stby(config)# sh cap capture asp-drop type asp-drop all circular-buffer [Capturing – 15441 bytes] match ip host 10.255.4.155 host 126.96.36.199
Opening tcpdump provides the following log:
ASA/sec/stby(config)# sh cap asp-drop
41: 12:18:20.000000 10.255.4.155.65256 > 188.8.131.52.22: S 4133841206:4133841206(0) win 64202 <mss 1366,nop,wscale 8,nop,nop,sackOK> Drop-reason: (no-adjacency) No valid adjacency
43: 12:18:22.998575 10.255.4.155.65256> 184.108.40.206.22: S 4133841206:4133841206(0) win 64202 <mss 1366,nop,wscale 8,nop,nop,sackOK> Drop-reason: (no-adjacency) No valid adjacency
46: 12:18:28.999444 10.255.4.155.65256 > 220.127.116.11.22: S 4133841206:4133841206(0) win 64202 <mss 1366,nop,wscale 8,nop,nop,sackOK> Drop-reason: (no-adjacency) No valid adjacency
Tcpdump states that we can’t ssh to our secondary unit because of Drop-reason: (no-adjacency). Cisco definition for this drop states no route to hostwhich is not the case since we do have reachability.
ASA/sec/stby(config)# sh debug debug icmp trace enabled at level 1
ICMP echo request from 10.255.4.155 to 18.104.22.168 ID=1 seq=188 len=32 ICMP echo reply from 22.214.171.124 to 10.255.4.155 ID=1 seq=188 len=32
Creating packet-tracer for this flow on stby unit provides interesting results:
Result: input-interface: mgmt_int input-status: up input-line-status: up output-interface: outside output-status: up output-line-status: up Action: drop Drop-reason: (fo-standby) Dropped by standby unit
With all the data and no valid issue based on my experience this smelled like Cisco Bug.
Little time on the call with Cisco I was provided with the following bug:
Based on the bug notes this is a Cisco limitation and there is no workaround for this behavior. Reason why I said at the beginning “may encounter” is because again this ONLY applies to anyconnect termination firewalls. Anything that you have southbound will be reachable as long as your ACL,NAT etc.. allows it thats it.
I hope this has been informative and feel free to drop a comment if you like it.
Bart is passionate about new technologies and their impact on our lives. He does not believe in titles or amount of certifications but positive attitude and motivation. Simply the guy that make things happen. You can reach him via Linkedin or meet him on CSGO. Currently focusing on architecting and designing custom-build hybrid cloud solutions around IaaS, DRaaS, BaaS realm.