AnyConnect Remote Access VPN

Troubleshooting and Best Practices

Paweł Cecot ([email protected])

CX Technical Leader
30.03.2020
Why this session?
• Due to COVID-19 global pandemic, more and more companies are
implementing remote working policies to prevent the spreading of
the disease.
• As a result, there is an increased demand for Remote Access VPN
(RAVPN) to provide employees with access to internal company
resources.
• We want to equip you with knowledge and tools to assist you with
managing and troubleshooting AnyConnect Remote Access VPNs.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 AnyConnect Overview
AnyConnect Configuration
AnyConnect Connection Flow
Agenda Troubleshooting Tools
Troubleshooting control plane issues
Troubleshooting data plane issues

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Session Objectives & Prerequisites
Session Objectives:
• The session will focus on Cisco AnyConnect Secure Mobility Client and
ASA as the VPN gateway.
• Understand the AnyConnect connection flow which is the key to
successful troubleshooting.
• Best practices for AnyConnect performance optimization.
• Troubleshooting techniques for typical control and data plane issues.

Prerequisites:
• Experience with ASA/FTD configuration and troubleshooting.
• Basic experience with AnyConnect configuration.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Panelist Q&A Guidelines
• On the panelist Q&A we will only be able to address general
questions.
• For questions that require in-depth analysis, we will ask to open a
TAC case.
• EMEAR TAC Security Workshop 2020.
• Session survey.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect Overview

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect as a remote access VPN solution
• Remote access VPN solution for desktops and mobile devices
• Allows users to access corporate resources from anywhere
• Uses TLS by default, but can be configured to use IPsec (IKEv2 only)
• Can be installed via a web connection, manually or through an App
Store.
Web Deploy Manual App Stores

Adaptive Security Identity Services

Appliance (ASA) Engine (ISE)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect for Desktop and Mobile Devices
End user perspective

AnyConnect Modules
• AnyConnect VPN
• AnyConnect Network Access Manager
• AnyConnect Web Security
• AnyConnect ISE Posture
• AnyConnect AMP Enabler
• Network Visibility Module
• Umbrella Roaming Security Module
• AnyConnect VPN Start Before Logon
• AnyConnect Diagnostic and Reporting Tool
• AnyConnect Posture (HostScan)
• AnyConnect Customer Experience Feedback

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Supported VPN Headends

Platform Protocols Minimum Version

ASA SSL & IPsec (IKEv2) 8.4(1)

FTD SSL & IPsec (IKEv2) 6.2.1

SSL - 15.1(4)M4
ISR G2 SSL & IPsec (IKEv2)
IPsec - 15.2(4)M
SSL - 3.12S
CSR1000v SSL & IPsec (IKEv2)
IPsec – 3.9S
ISR 4K, ISR 1K IPsec (IKEv2) 3.10S

ASR 1K IPsec (IKEv2) 3.5S

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect 4.x Licensing – Plus, Apex & VPN-Only
• Licenses are required to terminate RA VPN connections on a
headend.
• ASA platforms will only support 2 VPN peers without a license.
• FTD will not allow AnyConnect configuration to be deployed to the device without
licensing.

• The number of licenses needed for AnyConnect Plus or Apex is

based on all the possible unique users that may use RA VPN.
• VPN-Only is purchased based on the maximum number of
simultaneous connections and applied per individual firewall.
• For AnyConnect software download or technical support, the
contract must be linked to CCO ID.
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
A(1

AnyConnect Plus and Apex License Features

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How to verify licensing and scaling numbers?
ciscoasa# show vpn-sessiondb license-summary
---------------------------------------------------------------------------
VPN Licenses and Configured Limits Summary
--------------------------------------------------------------------------- With new licensing model, ASA
Status : Capacity : Installed : Limit
is always unlocked for its
-----------------------------------------
AnyConnect Premium : ENABLED : 750 : 750 : NONE maximum hardware capacity
AnyConnect Essentials : DISABLED : 750 : 750 : NONE
Other VPN (Available by Default) : ENABLED : 750 : 750 : NONE
[...]
---------------------------------------------------------------------------
VPN Licenses Usage Summary Peak In Use counter tells what
--------------------------------------------------------------------------- was the maximum concurrent
All : Peak : Eff. : connections
In Use : In Use : Limit : Usage
---------------------------------
AnyConnect Premium : : 50 : 94 : 750 : 6%
Anyconnect Client : : 50 : 90 : 750 : 6%
Other VPN : : 0 : 0 : 750 : 0%
L2TP Clients
---------------------------------------------------------------------------

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Problem #1 – unable to connect more users
• Customer had 25 licenses and upgraded to 250.
ciscoasa# show version | i AnyConnect|VPN
AnyConnect Premium Peers : 250 When license is installed, it will
AnyConnect Essentials : Disabled always show maximum
Other VPN Peers : 250 hardware capacity
Total VPN Peers : 250
AnyConnect for Mobile : Disabled
AnyConnect for Cisco VPN Phone : Disabled
ciscoasa# show running-config vpn-sessiondb
vpn-sessiondb max-anyconnect-premium-or-essentials-limit 25
ciscoasa#

%ASA-4-716023: Group name User user Session could not be established: session limit of maximum_sessions reached.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Problem #2 – unable to connect more users
Multi-Context Mode Remote-Access (AnyConnect) VPN

Allocate VPN Resources

class resource01 context context1
limit-resource VPN AnyConnect 900 member resource01
limit-resource VPN Burst AnyConnect 200 allocate-interface GigabitEthernet0/0
allocate-interface GigabitEthernet0/1
class resource02 config-url disk0:/context1
limit-resource VPN AnyConnect 900 join-failover-group 1
limit-resource VPN Burst AnyConnect 200

Verify VPN Resources

ciscoasa/pri/act# show resource usage all resource VPN AnyConnect
Resource Current Peak Limit Denied Context
AnyConnect 1 1 4000 0 context1
AnyConnect 1 1 4000 0 context2

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect Configuration

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
New AnyConnect Configuration – ASA (ASDM)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
New AnyConnect Configuration – FTD (FMC, FDM)
• Use FMC/FDM wizard:

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ASA SSL Remote Access VPN configuration
Local IP address pool.
ip local pool POOL_ASA1 192.168.1.100-192.168.1.200 mask 255.255.255.0
! Other options - AAA and DHCP
ssl trust-point ASDM_TrustPoint0 outside
! Certificate used by ASA during
webvpn TLS handshake
enable outside
anyconnect image disk0:/anyconnect-win-4.8.02045-webdeploy-k9.pkg 1
anyconnect enable Global WebVPN settings
tunnel-group-list enable
!
group-policy GroupPolicy_RA_VPN_SSL internal
group-policy GroupPolicy_RA_VPN_SSL attributes
dns-server value 10.0.10.100
vpn-tunnel-protocol ssl-client
#2 the connection first lands
default-domain value cisco.com on a tunnel-group
!
username pcecot password ***** pbkdf2
!
tunnel-group RA_VPN_SSL type remote-access
tunnel-group RA_VPN_SSL general-attributes
address-pool POOL_ASA1
#1 the connection first lands on a
default-group-policy GroupPolicy_RA_VPN_SSL tunnel-group
tunnel-group RA_VPN_SSL webvpn-attributes (group-url, group—alias, cert map)
group-alias RA_VPN_SSL enable
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ASA IPSec Remote Access VPN configuration
ip local pool POOL_ASA1 192.168.1.100-192.168.1.200 mask 255.255.255.0
! Dynamic Crypto Map
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set ikev2 ipsec-proposal AES256
crypto map outside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map outside_map interface outside
!
crypto ikev2 enable outside client-services port 443
IKEv2 enabled with client
crypto ikev2 remote-access trustpoint ASDM_TrustPoint0 services.
!
webvpn
anyconnect image disk0:/anyconnect-win-4.8.02045-webdeploy-k9.pkg 1
anyconnect profiles RA_VPN_client_profile disk0:/RA_VPN_client_profile.xml IPSec requires an XML
anyconnect enable profile.
tunnel-group-list enable
!
group-policy GroupPolicy_RA_VPN_IPSec internal
group-policy GroupPolicy_RA_VPN_IPSec attributes
vpn-tunnel-protocol ikev2
!
tunnel-group RA_VPN_IPSec type remote-access
tunnel-group RA_VPN_IPSec general-attributes
address-pool POOL_ASA1
default-group-policy GroupPolicy_RA_VPN
tunnel-group RA_VPN webvpn-attributes
group-alias RA_VPN enable
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Split Tunneling
• Split Tunneling - {tunnelall | tunnelspecified | excludespecified }
access-list SPLIT standard permit 10.0.0.0 255.255.255.0
group-policy GroupPolicy_SalesGroup attributes
split-tunnel-policy tunnelspecified
split-tunnel-network-list value SPLIT

• Dynamic Split Tunneling – split exclude based on DNS domain name

webvpn
enable outside
anyconnect-custom-attr dynamic-split-exclude-domains description dynamic-split-exclude-domains
!
anyconnect-custom-data dynamic-split-exclude-domains SAAS outlook.office.com,sharepoint.com,
outlook.office365.com
!
group-policy GroupPolicy_SalesGroup attributes
anyconnect-custom dynamic-split-exclude-domains value SAAS

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 NAT for Remote Access VPN
• NAT Exempt
nat (inside,outside) source static INSIDE INSIDE destination static AC_POOL AC_POOL no-proxy-arp route-lookup

• NAT Exempt for VPN Hairpinning (traffic between AC clients)

nat (ouside,outside) source static AC_POOL AC_POOL destination static AC_POOL AC_POOL no-proxy-arp route-
lookup
same-security-traffic permit intra-interface
On ASA required. On FTD not
applicable.
• PAT for the Internet access:
nat (ouside,outside) source dynamic AC_POOL dynamic

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
A(4

Anyconnect with ASA Failover

• Failover syncing does not replicate the following files:
• AnyConnect images
• HostScan images
• AnyConnect profiles

• After certificate enrolment perform “write standby” to replicate the RSA key
pair.
• AnyConnect is using Parent tunnel (control plane) and SSL/DTLS Tunnels
(data plane).
• Data plane tunnels are not stateful. After a failover those sessions need to
be re-established, which is done with the Parent tunnel and the “session
token”.
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect Connection Options

By IP Address

By Domain Name

By pre-defined profile

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Anyconnect Client Profile
• Contain a list of headends for users to choose from
• Contain configuration for advanced AnyConnect features
• Can be pushed down from the headend
• Stored on the client’s machine
• Windows: C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Profile\
• macOS: /opt/cisco/anyconnect/profile/
• Linux: /opt/cisco/anyconnect/profile/

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Anyconnect Client Profile Example
<?xml version="1.0" encoding="UTF-8"?>
<AnyConnectProfile xmlns="http://schemas.xmlsoap.org/encoding/">
<ClientInitialization>
<UseStartBeforeLogon UserControllable="true">false</UseStartBeforeLogon>
<AutomaticCertSelection UserControllable="true">false</AutomaticCertSelection>
<AutoConnectOnStart UserControllable="true">true</AutoConnectOnStart>
<AutoUpdate UserControllable="false">true</AutoUpdate>
</ClientInitialization>
<ServerList>
<HostEntry>
<HostName>ciscoasa (IPsec) IPv4</HostName>
<HostAddress>172.16.221.130</HostAddress>
<PrimaryProtocol>IPsec</PrimaryProtocol>
</HostEntry>
</ServerList>
</AnyConnectProfile>

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect Connection Flow

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect Connection Flow

TLS or IKEv2

Pre-Authentication
Authentication
Group-Policy assignment
HostScan (optional)

Client Services

Tunnel establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Aggregate Authentication and Configuration
• Aggregate Authentication and Configuration is a proprietary protocol
used by AnyConnect.
• The protocol is transport agnostic and can be used both with TLS
and IKEv2.
• It defines a common XML format which is used when authenticating
and configuring the client during the tunnel establishment.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
Pre-Authentication
AnyConnect
•

Authentication
Connection
•

Client Services
Flow
•

• Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
TLS 1.2 Handshake
Client Server
TCP Handshake
Client Hello
[TLS Version, Random #, Ciphers, Compression, Extensions]

Server Hello
[Chosen TLS Version, Random #, Chosen Ciphers, Compression, Extensions
[Certificate]
[ServerKeyExchange]
ServerHelloDone
ClientKeyEchange
ChangeCipherSpec
Finished
ChangeCipherSpec
Finished
Application Data
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
IKEv2 Exchange
1. Cisco AnyConnect client initiates IKEv2 connection to the VPN headend.
2. In addition to typical IKE payloads, the client sends vendor ID payloads to indicate
support for Fragmentation, Redirect, Cisco HostScan, AnyConnect EAP.
3. The Aggregate Authentication and Configuration is carried over EAP.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
Pre-Authentication
AnyConnect
•

Authentication
Connection
•

Client Services
Flow
•

• Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
HostScan / Pre-Login (Optional)
• If HostScan (posture module) is
used, it will send certain parameters
to ASA.
• Avalable only on AnyConnect 3.x.
• Deprecated on AnyConnect 4.x+.
• Failing a scan prior to authentication
means that there is an issue with the
pre-login policy, NOT the dynamic
access policies

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Certififcate Tunnel-Group Mapping (Optional)
• Tunnel-group decides what authentication method is used which is
why this step occurs pre-auth.
• This will happen only with the following configuration:
webvpn
certificate-group-map RA_CERT_MAP 10 RA_VPN_SSL_RSA Tunnel-group name
!
crypto ca certificate map RA_CERT_MAP 10
issuer-name co cn = subca
!
tunnel-group-map enable rules

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
Pre-Authentication
AnyConnect
•

Authentication
Connection
•

Client Services
Flow
•

• Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Authentication
• User receives an authentication prompt or automatically sends a cert
• Failures at this level usually indicate…
1. Communication with the authentication server has failed
• Packets lost / dropped
• Failed password exchange with server
• Server not configured correctly
2. The user’s login credentials are genuinely incorrect
• By default, authentication must complete in 12 seconds
(configurable)
• If the attempt takes longer, determine the cause of the latency
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
License Approval
• At this stage, the total remaining license count is decremented by 1
• If there are 0 remaining licenses, the connection is terminated
• Users will only see “Login Failed” – nothing about licenses

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
HostScan (Optional)
• HostScan sends all of the data it collected during the pre-auth
phase
• ASA evaluates the user’s settings against its dynamic access
policies
• Can terminate sessions not meeting certain criteria
• Can apply new settings or ACLs to users based on certain criteria

• If the user is failing posture checks, ensure they are matching the
correct DAP

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Group-Policy Assignment
• The user is placed into a group-policy one of two ways
1. The ‘default-group-policy’ command under the tunnel-group
2. The RADIUS/LDAP server maps the user to a group-policy
• The group-policy defines the XML profiles, modules, etc. to push
down to the user during the next phase.
• Failures at this stage suggest a problem with the attributes in the
group-policy (e.g. ‘simultaneous-logins’ is set to 0)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect Modules Versions and Profile check.
In this step, the client receives:
• AnyConnect modules versions, which are used by the VPN
Downloader in the next-section.
• AnyConnect profile hash.
• If the client does not have the profile, the VPN downloader in the next
section downloads this profile.
• If the client has the profile already, the sha-1 hash of the client-profile is
compared with that of the server. In case of mismatch, the client profile
is overwritten with server’s profile.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
Pre-Authentication
AnyConnect
•

Authentication
Connection
•

Client Services
Flow
•

• Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
File Pushing
• The ASA can push down several types of files
• Client profiles
• GUI language localization
• GUI customization
• AnyConnect will terminate at this stage if any of these files fail to
install due to permission issues, registry problems, etc.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Client Upgrade
• If the ASA’s .pkg version of AnyConnect is newer than the local
version, the client will be upgraded automatically
• If the ASA’s .pkg version of AnyConnect is older than the local
version, the client will stay the same (it will NOT downgrade)
• Failing at this stage indicates issues with the upgrade/installation
process.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
SCEP Proxy (Optional)
• AnyConnect users can pull a user certificate from an internal server
via SCEP proxy
• Failing at this stage usually indicates…
1. The ASA could not communicate with the certificate authority
2. The client was unable to pull a certificate (key-size too small, etc.)
3. The cert retrieved is invalid for AnyConnect (e.g. no client-auth EKU)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
Pre-Authentication
AnyConnect
•

Authentication
Connection
•

Client Services
Flow
•

• Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Establishing VPN! I’m Clear!
…or not. Most users assume that once they see this message,
everything else is already completed.
There are actually a few remaining steps the client goes through.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
IP Address Assignment
• IPs can be assigned from a local pool, the AAA server, or DHCP
• All three can cause the client to fail establishment as an IP is
required
• Local pool – can run out of available addresses
• AAA server – may not be properly configured to assign an IP
• DHCP server – may not be responding to the ASA

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MTU Adjustment
• The MTU for AnyConnect is calculated based on the size the ASA
believes it needs to be to avoid fragmentation.
• If the MTU is calculated incorrectly due to headers it is not
expecting, or the client’s network adapter rejects the MTU setting,
the connection will terminate.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
TLS/DTLS Data Tunnel Establishment
• The AnyConnect will try to bring two data tunnels – TLS and DTLS.
DTLS being the preferred one (best performance).
• To bring TLS data tunnel, AnyConnect will use the HTTP CONNECT
method.
• Next, the AnyConnect tries to bring the DTLS data tunnel, however
at this stage, the traffic is already using TLS tunnel.
• It is only when the DTLS handshake completes, a seamless
transition is made to DTLS data encryption channel.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Troubleshooting Tools
AnyConnect

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConect Message History

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
DART - Diagnostics and Reporting Tool
macOS Windows

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
DART - Diagnostics and Reporting Tool
• Creates a zip file, e.g. DARTBundle_0324_2116.zip

Windows Local Logs

(Event Viewer)

XML Profile
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
”Live logs” - Windows

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
”Live logs” – macOS / Linux

Linux: /var/log/syslog
NOTE: The logging location on each Linux distribution can be different
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Analysing DART logs
• Different connection requests can be split by:
An IPsec VPN connection to <> has been requested by the user.
An SSL VPN connection to <> has been requested by the user.

• Compare logs for working and not working scenario.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Packet Capture on client device.

Physical adapter
(Ethernet, Wi-Fi) - traffic
on the wire.

AnyConnect adapter – traffic before encryption.

Not available until connection goes up.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Troubleshooting Tools
ASA / FTD

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Logging Facility Preparation and Best Practices
• Use Network Time Protocol (NTP) in order to sync the clock between all
devices that are debugged.
• Logging level debugging (severity=7) or informational (severity=6)
logging enable
logging timestamp
logging buffer-size 1000000
logging buffered debugging

• Clear the buffer shortly before reproducing the problem

ciscoasa/pri/act# clear logging buffer

• If there are many sessions, use an external syslog server.

• Syslog messages are very valuable but very often forgotten.
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Debugs on ASA and FTD
• By default, debugs on ASA and FTD are printed on
console/ssh/telnet session.
• If the debugs are too chatty, we can redirect them to a syslog
message 711001:
ciscoasa(config)# logging debug-trace
INFO: 'logging debug-trace' is enabled. All debug messages are currently
being redirected to syslog:711001 and will not appear in any monitor session

• Debugs are stopped as soon as the console/ssh/telnet session is

closed, hoever we can enable persistent debugging:
ciscoasa(config)# logging debug-trace persistent

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Packet-tracer
• The packet-tracer command provides detailed information about the
packets and how they are processed by the ASA.
• It injects a virtual packet into the security appliance and track the
flow from ingress to egress.
• With VPN’s, running a packet-tracer in a reverse direction could be
helpful with verifying features such as NAT, ACL, routing, etc.
ciscoasa# packet-tracer input inside tcp 10.0.0.5 80 192.168.1.100 12345 detailed

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Packet Capture on ASA/FTD
• The recommend way is to configure captures using CLI.
Directions does not matter as it
• The syntax is very flexible and easy to use: is bi-directional.

ciscoasa# capture <capture_name> interface <interface_name> match <ACL type syntax>

• Useful capture options:

• trace detail - provides a packet-tracer output for real traffic.
• type isakmp - capture encrypted and decrypted IKEv1/IKEv2 payloads.
• include-decrypted – include IPsec/SSL VPN decrypted packets in the
capture (inbound traffic only); ASA 9.9(1)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect
Troubleshooting control
plane issues

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect Troubleshooting Methodology
• The key is to be able to isolate the problem to a specific feature,
subsystem, or component.
• It is good to start with the least intrusive methods so that the production
environment is not negatively impacted.
• The key is to break the problem down to one of the following:
Control Plane
• Connectivity issues, TLS handshake, Authentication, Authorization,
session disconnects, etc.

Data plane
• Traffic not flowing or impacted, high CPU, performance issues, etc.
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Before you start debugging…
Understand and isolate the problem
• Was the AC connection/traffic working before?
• What is the business impact, how many users are impacted?
• Are all of the users affected or only specific one?
• Ask for screenshots of the potential Anyconnect error messages.
• At what “stage” AnyConnect is failing?
• Do you see username/password prompt? (TCP/TLS, IKEv2, prior-
authentication)
• Authentication
• Does it fail after a user prompt? (client services)
• After client services, before establishment. (tunnel establishment)
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
• Pre-Authentication
Troubleshooting • Authentication
control plane • Client Services
issues • Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 TLS - TCP Handshake
Packet Capture

Syslogs
%ASA-6-302013: Built inbound TCP connection 1338 for outside:172.16.221.1/52139 (172.16.221.1/52139) to
identity:172.16.221.130/443 (172.16.221.130/443)

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Problem #1 - TCP Handshake

SYN or SYN/ACK was blocked in transit

Troubeshooting steps:
• Collect packet capture on both client and VPN gateway

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Problem #2 - TCP Handshake
The webvpn service is not enabled or not listening on the
VPN gateway
Troubleshooting steps:
• Verify the configuration („show running-config webvpn”)
webvpn
enable outside
anyconnect enable

• Verify if the ASA/FTD is listening on the proper socket

ciscoasa# show asp table socket
Protocol Socket State Local Address Foreign Address
SSL 00e67108 LISTEN 172.16.221.130:443 0.0.0.0:*
DTLS 00e69238 LISTEN 172.16.221.130:443 0.0.0.0:*
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 TLS 1.2 Handshake – possible problems
• Not receiving a “Server Hello” usually indicates that the ASA has a problem loading its
identity certificate (e.g. no private key)
• Failing immediately after the “Server Hello” usually indicates a problem with the
certificate’s attributes (no “server-auth”, no CN in the subject-name, etc.)
Troubeshooting steps – syslogs and captures
%ASA-6-725001: Starting SSL handshake with client outside:192.168.20.1/52715 for TLS session.
%ASA-7-725010: Device supports the following 3 cipher(s).
%ASA-7-725011: Cipher[1] : AES256-SHA
%ASA-7-725011: Cipher[2] : DES-CBC3-SHA
%ASA-7-725011: Cipher[3] : DHE-RSA-AES256-SHA
%ASA-7-725008: SSL client outside:192.168.20.1/52715 proposes the following 8 cipher(s).
%ASA-7-725011: Cipher[1] : AES256-SHA
%ASA-7-725011: Cipher[2] : AES128-SHA
%ASA-7-725011: Cipher[3] : DHE-DSS-AES256-SHA
%ASA-7-725011: Cipher[4] : DHE-DSS-AES128-SHA
%ASA-7-725011: Cipher[5] : DES-CBC3-SHA
%ASA-7-725011: Cipher[6] : EDH-DSS-DES-CBC3-SHA
%ASA-7-725011: Cipher[7] : RC4-SHA
%ASA-7-725011: Cipher[8] : RC4-MD5
%ASA-7-725012: Device chooses cipher : AES256-SHA for the SSL session with client outside:192.168.20.1/52715
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
%ASA-6-725002: Device completed SSL handshake with client outside:192.168.20.1/52715
%ASA-6-725007: SSL session with client outside:192.168.20.1/52715 terminated.
IKEv2 Exchange
Many possible problems at this stage. Troubleshooting
similar to any other IKEv2 VPN.
Troubleshooting steps:
• Debugs
ciscoasa# debug crypto condition peer <client’s public IP>
ciscoasa# debug crypto ikev2 protocol 254
ciscoasa# debug crypto ikev2 platform 254

• ISKAMP type capture

ciscoasa# capture <capture name> type isakmp interface <interface name>

• Syslogs
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
• Pre-Authentication
Troubleshooting • Authentication
control plane • Client Services
issues • Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Certificate Mapping Issues
• If the SSL Handshake completes, but no login prompt appears, run ‘show run
webvpn’ on the ASA to see if any cert maps exist:
ciscoasa(config-webvpn)# sh run webvpn | i certificate-group-map
certificate-group-map RA_CERT_MAP 10 RA_VPN

• Check syslog messages:

Mar 29 2020 13:38:07: %ASA-7-717036: Looking for a tunnel group match based on certificate maps for
peer certificate with serial number: 25BC641A0237030A, subject name: CN=win10, issuer_name: CN=SubCA.
Mar 29 2020 13:38:07: %ASA-4-717037: Tunnel group search using certificate maps failed for peer
certificate: serial number: 25BC641A0237030A, subject name: CN=win10, issuer_name: CN=SubCA.

• Debugs prior 9.13(1):

ciscoasa# debug crypto ca
ciscoasa# debug crypto ca messages
ciscoasa# debug crypto ca trsansactions

• Debugs 9.13(1):
ciscoasa# debug crypto ca 8

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
• Pre-Authentication
Troubleshooting • Authentication
control plane • Client Services
issues • Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Problems related with AAA server (1/2)
• Reachability to the AAA server.
Troubleshooting steps:
• “show run tunnel-group” to determine the AAA server being used
• “show run aaa-server” and “show aaa-server” to verify configuration and
state.
• Use the ”test aaa-server” to verify the AAA server:
ciscoasa/pri/act# test aaa-server authentication ISE host 10.23.10.123 username pcecot
Password: *********
INFO: Attempting Authentication test to IP address (1.1.1.1) (timeout: 12 seconds)
ERROR: Authentication Server not responding: No active server found

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Problems related with AAA server (2/2)
• Sometimes checking AAA server logs is faster end more efficient in
determining the root cause.
• Incorrect Radius shared secret.
• User rejected.
• Wrong authorization rule matched.

• Capture between AAA server and VPN headend.

• Debugs:
• debug ldap 255
• debug radius all

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
• Pre-Authentication
Troubleshooting • Authentication
control plane • Client Services
issues • Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Disable AnyConnect downloader
• As a test disable the AnyConnect downloader.
• The client services downloader can be disabled via the local policy
• Windows: C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility
Client\AnyConnectLocalPolicy.xml
• macOS & Linux: /opt/cisco/anyconnect/AnyConnectLocalPolicy.xml

• Change “BypassDownloader” from “false” to “true”

• Try connecting again – if it works, client services is the problem

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Client Services Troubleshooting
• Verify read/write permissions to C:\ProgramData\Cisco\Cisco
AnyConnect Secure Mobility Client
• Review the DART logs.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • TLS or IKEv2
• Pre-Authentication
Troubleshooting • Authentication
control plane • Client Services
issues • Tunnel Establishment

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 IP address assignment
• Check syslog messages:
Mar 29 2020 15:47:03: %ASA-7-737035: IPAA: Session=0x00011000, 'IPv4 address request' message queued
Mar 29 2020 15:47:03: %ASA-7-737035: IPAA: Session=0x00011000, 'IPv6 address request' message queued
Mar 29 2020 15:47:03: %ASA-7-737001: IPAA: Session=0x00011000, Received message 'IPv4 address request'
Mar 29 2020 15:47:03: %ASA-5-737003: IPAA: Session=0x00011000, DHCP configured, no viable servers found for
tunnel-group 'RA_VPN_SSL'
Mar 29 2020 15:47:03: %ASA-4-737019: IPAA: Session=0x00011000, Unable to get address from group-policy or tunnel-
group local pools
Mar 29 2020 15:47:03: %ASA-5-737007: IPAA: Session=0x00011000, Local pool request failed for tunnel-group
'RA_VPN_SSL'
Mar 29 2020 15:47:03: %ASA-4-737012: IPAA: Session=0x00011000, Address assignment failed
Mar 29 2020 15:47:03: %ASA-3-722020: TunnelGroup <RA_VPN_SSL> GroupPolicy <GroupPolicy_RA_VPN_SSL> User <cisco>
IP <172.16.221.1> No address available for SVC connection

• Debug RADIUS, DHCP.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
DTLS Failure
• When DTLS fails, it should fall back gracefully to TLS
• Perform a packet capture specifically for UDP 443 traffic on both the
client and the ASA – verify full establishment
• Disable DTLS in the group-policy and try again
group-policy groupPolicyName attributes
webvpn
anyconnect ssl dtls none

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Troubleshooting data
plane issues

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Before you start debugging…
Underetsand and isolate the problem
• Was it working before? If yes, were there any changes on the
client/headend but also on other network devices?
• Determine what application is affected (protocol/port)?
• Is application not reachable at all or partially affected?
• What is the source IP (typically AC Client IP) and destination IP
(server, other VPN user, etc.)?
• What is the ‘username’?
• Clarify what other network devices are present between the VPN
gateway (ASA, Router) and the destination. This will tell you if there
are some other FW, LB etc. that may have impact on the traffic?
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • Unreachable Resources
Troubleshooting over the Tunnel
data plane issues • High CPU caused by
VPN

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Identify the connection parameters.
ciscoasa/pri/act# sh vpn-sessiondb anyconnect filter name pcecot Username
Session Type: AnyConnect

Username : pcecot Index : 25964

Assigned IP : 192.168.1.100 Public IP : 172.16.221.1
Protocol : IKEv2 IPsecOverNatT AnyConnect-Parent
License : AnyConnect Premium
Encryption : IKEv2: (1)AES256 IPsecOverNatT: (1)AES256 AnyConnect-Parent: (1)none
Hashing : IKEv2: (1)SHA1 IPsecOverNatT: (1)SHA1 AnyConnect-Parent: (1)none
Bytes Tx : 0 Bytes Rx : 0
Group Policy : GroupPolicy_RA_VPN Tunnel Group : RA_VPN
Login Time : 03:14:38 UTC Wed Mar 25 2020 Tunnel-group and group-policy
Duration : 0h:00m:32s used by a given user.
Inactivity : 0h:00m:00s
VLAN Mapping : N/A VLAN : none
Audt Sess ID : 0a0000010656c0005e7acc9e
Security Grp : none

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Collect packet captures - 1/3

outside inside

Anyconnect user ASA Server IP: 10.0.0.1

PHY IP: 172.16.221.128 outside IP: 172.16.254.1
AC IP: 192.168.1.100 inside IP: 10.0.0.254

Step1 – collect capture on ASA inside “interface”

ciscoasa# capture IN interface inside trace detail match icmp host 192.168.1.100 host 10.0.0.1
ciscoasa# ! generate the traffic
ciscoasa#
ciscoasa# show capture IN

0 packet captured

0 packet shown
ciscoasa#

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Collect packet captures - 2/3
Step 2 – collect capture on ASA’s “outside” interface with “include-
decrypted” option
ciscoasa# capture AC type raw-data trace detail interface outside include-decrypted match icmp host
192.168.1.100 host 10.0.0.1
ciscoasa/pri/act# ! generate the traffic
ciscoasa/pri/act#
ciscoasa/pri/act# show capture AC

4 packets captured

1: 22:17:14.314116 192.168.1.100 > 10.0.0.1 icmp: echo request

2: 22:17:18.966228 192.168.1.100 > 10.0.0.1 icmp: echo request
3: 22:17:23.966334 192.168.1.100 > 10.0.0.1 icmp: echo request
4: 22:17:28.982493 192.168.1.100 > 10.0.0.1 icmp: echo request
4 packets shown

BINGO! Was it dropped or routed via a different interface?

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Collect packet captures - 3/3
Thanks to “trace detail” option we can check what happened:
ciscoasa# show capture AC packet-number 1 trace detail
(…)
Phase: 9
Type: ACCESS-LIST
Subtype: filter-aaa
Result: DROP
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7f10f51f0990, priority=13, domain=filter-aaa, deny=true
hits=5, user_data=0x7f10de499a00, filter_id=0x3(ICMP), protocol=1
src ip=0.0.0.0, mask=0.0.0.0, icmp-type=0
dst ip=10.0.0.1, mask=255.255.255.255, icmp-code=0

Result:
input-interface: outside
input-status: up
input-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000055f3f95bc377
flow (acl-drop)/snp_sp_action_cb:1788
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Root cause?
• VPN-Filter was dropping the traffic.
ciscoasa# show vpn-sessiondb detail anyconnect filter name cisco | i Filter
Filter Name : ICMP
Filter Name : ICMP
ciscoasa# show access-list ICMP
access-list ICMP; 2 elements; name hash: 0x2d2cf426
access-list ICMP line 1 extended deny icmp any host 10.0.0.1 (hitcnt=8) 0x0454eb75
access-list ICMP line 2 extended permit ip any any (hitcnt=0) 0x3ec775f0
ciscoasa#
ciscoasa# sh running-config group-policy GroupPolicy_RA_VPN_SSL
group-policy GroupPolicy_RA_VPN_SSL internal
group-policy GroupPolicy_RA_VPN_SSL attributes
wins-server none
dns-server none
vpn-filter value ICMP
vpn-tunnel-protocol ssl-client
default-domain none

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Problem #2
Phase: 14
Type: NAT
Subtype: rpf-check
Result: DROP
Config:
nat (inside,outside) source dynamic INSIDE interface
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7f10f543a0b0, priority=6, domain=nat-reverse, deny=false
hits=1, user_data=0x7f10f55cdc70, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=10.0.0.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=inside

Result:
input-interface: outside
input-status: up
input-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule, Drop-location: frame 0x000055f3f95bc377
flow (nat-rpf-failed)/snp_sp_action_cb:1140
Missing NAT exempt for
ciscoasa/pri/act# sh run nat
nat (inside,outside) source dynamic INSIDE interface
Anyconnect traffic
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
nat (inside,outside) 1 source static INSIDE INSIDE destination static AC_POOL AC_POOL no-proxy-arp route-lookup
 • Unreachable Resources
Troubleshooting over the Tunnel
data plane issues • High CPU caused by
VPN

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
1 - Check for high CPU utilization
ciscoasa# show processes cpu-usage non-zero
PC Thread 5Sec 1Min 5Min Process
0x00000000019da592 0x00007fffd808b040 0.0% 0.0% 0.5% Logger
0x0000000000844596 0x00007fffd807bd60 0.0% 0.0% 0.1% CP Processing
0x0000000000c0dc8c 0x00007fffd8074960 0.1% 0.1% 0.1% ARP Thread
- - 43.8% 43.8% 40.3% DATAPATH-0-2209
- - 43.9% 43.8% 40.3% DATAPATH-1-2210

ciscoasa# show cpu usage

CPU utilization for 5 seconds = 88%; 1 minute: 88%; 5 minutes: 82%

• In this case, the ASA is oversubscribed.

• Need to check if this is related to a large amount of encrypted and
decrypted traffic.
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
2 - Calculate the crypto throughput
ciscoasa# clear crypto accelerator statistics.
ciscoasa# show clock
ciscoasa# show crypto accelerator statistics

Crypto Accelerator Status

-------------------------
[Capability]
Supports hardware crypto: False
Supports modular hardware crypto: False 𝐼𝑛𝑝𝑢𝑡 𝑏𝑦𝑡𝑒𝑠 + 𝑂𝑢𝑡𝑝𝑢𝑡 𝑏𝑦𝑡𝑒𝑠 ∗ 8
Max accelerators: 1 𝑀𝑏𝑝𝑠
Max crypto throughput: 225 Mbps 𝑇1 − 𝑇0
Max crypto connections: 250
[Global Statistics]
Number of active accelerators: 1
Number of non-operational accelerators: 0 Note 1 – counters will increase for all
Input packets: 1186 encrypted/decrypted traffic (HTTPS, SSL, IPsec,
Input bytes: 77360 SSH, etc.)
Output packets: 686
Note 2 - CSCvt46830 - FPR2100 'show crypto
x2
Output error packets: 0
Output bytes: 302496 accelerator statistics' counters do not track
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
symmetric crypto
3 - Benchmark against the datasheet

“Performance will vary depending on

features activated, and network traffic
protocol mix, and packet size
characteristics. Performance is
subject to change with new software
releases. Consult your Cisco
representative for detailed sizing
guidance.”

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 • Scale the RA deployment
• Split Tunneling
• Exclude SaaS – Webex, O365
AnyConnect Per-App VPN (Mobile
Potential Mitigations
•
Devices)
for High CPU caused • Tunnel Protocol Selection –
by VPN TLS/DLTS/IPSec
• Crypto Engine Accelerator Bias (ASA)
• Rate-Limit Traffic per AnyConnect User
• AnyConnect Performance Optimization
Tips

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
#1 – Scale the RA deployment
• ASA Load balancing is a mechanism for equitably distributing remote
access VPN traffic among the devices in a virtual cluster.
• https://www.cisco.com/c/en/us/td/docs/security/asa/asa913/config
uration/vpn/asa-913-vpn-config/vpn-ha.html
• Other options:
o DNS load balancing
o Anycast — distribute the same IP
o Hardware or software load-balancers

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
#2 - Split Tunneling
• By default, all traffic is sent to the ASA.
access-list SPLIT standard permit 10.0.0.0 255.255.255.0
group-policy GroupPolicy_SalesGroup attributes
split-tunnel-policy tunnelspecified
split-tunnel-network-list value SPLIT

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
#3 - Optimize Anyconnect Split Tunnel for
Office365 - 1/2
• Exclude traffic destined to Microsoft Office 365 and Cisco Webex
from a VPN connection.
• It incorporates network address exclusions and dynamic (FQDN
based) exclusions for Anyconnect clients that support it.
https://www.cisco.com/c/en/us/support/docs/security/anyconnect-secure-
mobility-client/215343-optimize-anyconnect-split-tunnel-for-off.html

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
#3 - Optimize Anyconnect Split Tunnel for
Office365 - 2/2

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
#4 - AnyConnect Per-App VPN (Mobile Devices)
AnyConnect Per-Application VPN (PerApp VPN) provides VPN
support to AnyConnect on mobile devices where tunnelling only
applications defined by a policy to the corporate network is desired.
Jabber

Salesforce

Facebook

LinkedIn
ASA

https://community.cisco.com/t5/security-documents/anyconnect-
per-app-vpn/ta-p/3646866
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
#5 - Tunnel Protocol Selection – TLS/DLTS/IPSec
• To achive best perfromance use IPSec or DTLS.
• If UDP 443 traffic is blocked between the VPN headend and the
AnyConnect client, it will automatically fallback to TLS
ciscoasa/pri/act# show vpn-sessiondb
[…]
Active : Cumulative : Peak Concurrent
----------------------------------------------
IKEv2 : 0 : 6 : 1
IPsecOverNatT : 0 : 6 : 1
AnyConnect-Parent : 2 : 10 : 2
SSL-Tunnel : 2 : 8 : 2
DTLS-Tunnel : 1 : 5 : 1
---------------------------------------------------------------------------
The more equal the better
Totals : 5 : 35 (there always need to be an SSL-Tunnel)
---------------------------------------------------------------------------

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
#6 - Implement Crypto Engine Accelerator Bias
(ASA Only)
• Crypto Engine Accelerator Bias is used to reallocate the crypto
cores to favour one encryption protocol over the other (SSL or
IPsec).
ciscoasa (config)# crypto engine accelerator-bias ?

configure mode commands/options

balanced - Equally distribute crypto hardware resources
ipsec-client - Allocate crypto hardware resources to favor IPsec/Encrypted Voice (SRTP)
ssl-client - Allocate crypto hardware resources to favor SSL

Note1: This command causes traffic disruption to services that require crypto operations.
Note2: Cryptographic core rebalancing is available on the following platforms: ASA 5585, 5580,
5545/5555, 4110, 4120, 4140, 4150, SM-24, SM-36, SM-44 and ASASM.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
#7 - Rate-Limit Traffic per AnyConnect User
• For FTD this is possible:
https://www.cisco.com/c/en/us/td/docs/security/firepower/640/configuration/guide/fp
mc-config-guide-v64/firepower_threat_defense_remote_access_vpns.html

• For ASA the only option as of now is to do that per Tunnel Group:
https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-
generation-firewalls/82310-qos-voip-vpn.html
https://www.cisco.com/c/en/us/td/docs/security/asa/asa913/configuration/firewall/asa
-913-firewall-config/conns-qos.html

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
AnyConnect Performance Optimization Tips
• AnyConnect 4.8.x and DTLS v1.2 or IKEv2 for the headend (FTD 6.6/ASA 9.10+) configuration .
• Verify the optimization setting for crypto hardware (Crypto Engine Accelerator Bias)
• Cipher Suite: Ideally, the AES-GCM will provide the best performance results.
• MTU configuration on the Group Policy: Ideally the higher the better, never exceeding 1406 →
Test.
• (ASA) AnyConnect tunnel optimizations can be enabled on ASA devices to potentially optimize
throughput available per client.
webvpn
anyconnect-custom-attr TunnelOptimizationsEnabled description Optimizations Enabled
anyconnect-custom-data TunnelOptimizationsEnabled False false
anyconnect-custom-data TunnelOptimizationsEnabled True true
!
group-policy <Group Policy Name> attributes
anyconnect-custom TunnelOptimizationsEnabled value True
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Conclusion
• Have a clear and concise problem description
• Understand the expected protocol flow on the control plane so that
the proper component is debugged
• Understand where/how to checkpoint the data plane
• There is always TAC!

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Useful resources

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Cisco CLI Analyzer

https://cway.cisco.com/go/sa/

• The Cisco CLI Analyzer is a smart SSH/Telnet client designed to help troubleshoot and
check the overall health of Cisco devices.
• Supports IOS, IOS-XE, IOS-XR, ASA, NX-OS, FXOS, AireOS (WLC), StarOS, ACI-OS, AP-
COS, APIC, ISE, VxWorks with planned future expansions.

• Contextual Help & Highlighting.

• Integrated TAC Tools (Traceback Analyzer, Firewall Top Talkers, etc.).

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
System Diagnostics - Check your system’s health

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Packet Capture Config Generator and Analyzer

https://cway.cisco.com/tools/CaptureGenAndAnalyse/

• The tool is designed to aid in collection and analysis of network packet captures taken on
IOS, IOS-XE and ASA/FTD devices.

• It allows to covert a hex dump output to a pcap format.

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Recently published articles on cisco.com
• Obtaining an Emergency COVID-19 AnyConnect License
https://www.cisco.com/c/en/us/support/docs/security/anyconnect-secure-mobility-client/215330-obtaining-an-emergency-covid-19-anyconne.html

• AnyConnect VPN, ASA, and FTD FAQ for Secure Remote Workers
https://www.cisco.com/c/en/us/td/docs/security/asa/misc/anyconnect-faq/anyconnect-faq.html

• AnyConnect Implementation and Performance/Scaling Reference for COVID-19 Preparation

https://www.cisco.com/c/en/us/support/docs/security/anyconnect-secure-mobility-client/215331-anyconnect-implementation-and-performanc.html

• How to optimize AnyConnect for Microsoft Office365 and Cisco WebEx connections
https://www.cisco.com/c/en/us/support/docs/security/anyconnect-secure-mobility-client/215343-optimize-anyconnect-split-tunnel-for-off.html

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Confidential