Study
Study
Issues
Cisco IOS XE running on Cisco Catalyst 9800 WLCs is essentially
composed of a Linux Kernel, with Cisco IOS and all the wireless processes
implemented as daemons. All the process daemons can be bundled under
the generic term "control plane." They are responsible for the following,
which are destined to and from the Cisco Catalyst 9800 Series WLC:
• CAPWAP
• Mobility
• Radio Resource Management (RRM)
• Rogue management
• Network Mobility Service Protocol (NMSP)
The Cisco tools enable you to gather information from the equipment
directly in your wireless environment while facilitating some of the most
common tasks you would take during troubleshooting. Thus, these tools
can be used for different purposes, such as troubleshooting AP issues,
client connectivity issues, mobility, and so on.
The Cisco wireless troubleshooting tools can be grouped into the following
categories:
You can use the Cisco WLC built-in tools to see the APs' status and
detect APs that have join issues. Furthermore, you can observe the syslog
and benefit from the trace-on-failure, always-on tracing, and radioactive
tracing features to easily find the failure reason while narrowing your
investigation to specific log events related to the APs that are experiencing
issues. Suppose you still cannot resolve the issue. In that case, you can
leverage the packet-capturing facility in the controller and perform a
deeper analysis of the packets destined to, sourced from, and passing
through the Cisco 9800 WLCs.
When you are faced with an AP issue in your environment, you can also
speed up the troubleshooting process if you are familiar with how to use
the Cisco standalone tools. Besides different Cisco tools that can be used
for monitoring of the wireless environment, as well as to analyze
information from the wireless devices or perform RF spectrum analysis,
some of them can be beneficial from AP issues troubleshooting as well.
For example, you can use Cisco available tools to perform common tasks
such as analyzing the Cisco WLC configuration, inspecting WLC debug
logs in a structural manner, and so on.
The Dashboard page in the GUI represents a single pane view that you
may want to look at daily and when troubleshooting issues since it
provides an overview of the wireless network status. The following figure
depicts the information provided by the dashboard.
Thus, you can compare your baseline state with the current state of your
network while observing the WLANs and APs (how many of them are up
and down), active and excluded clients, rogue APs and clients, and so on.
The Cisco Catalyst 9800 Series Wireless Controller has specific built-in
troubleshooting features that help you quickly identify root causes for
wireless issues in your network with less time and effort. In the GUI, you
can navigate to the Troubleshooting menu to access different tools for
control plane and data plane tracing.
These troubleshooting tools enable you to perform a root cause analysis
while following a systematic approach. For example, during a typical client
connectivity issue, while knowing when the issue was reported, you can
inspect details about failed events in the system, including events that
have occurred in the past, even without having any debugs enabled. If this
action does not help, you can perform an in-depth analysis by
conditionally increasing the level of logging per different processes,
performing packet capture, and so on.
Syslog
You can use the logs generated by the Cisco 9800 Series Wireless
Controller as the first means to verify the general health of the system. Any
violations of the predefined threshold for system resources like CPU,
memory, and buffers are reported into the log. In addition, any errors
generated by any subsystems get written in the logs. You will be able to
view the syslog server messages from the APs as well after they join the
controller. Thus, the syslog can be useful for troubleshooting AP join
issues. To view the logs, navigate to Troubleshooting > Syslog.
The syslog server on the APs and WLC has many levels and facilities.
While the level defines the severity, the facility enables you to specify
where you can put the syslog messages, for example, the syslog itself.
You can leverage a syslog server for an AP logs, which is applied through
an AP join profile once the AP is associated with the WLC, using the
following commands:
You can configure the syslog server for the WLC using the following
commands:
Trace-on-Failure
The Cisco 9800 Series Wireless Controller can keep track of predefined
failure conditions. There is nothing required for this feature to work. It is
continuously working in the background without the need for any debug
command. Hence, you can view the different failure conditions detected
and the number of events, with details about failed events. This feature
enables you to be proactive and detect issues that could be occurring in
your network, even without clients reporting them.
The following example shows the output that you can get for failure
conditions detected in the last two hours and details about the events:
-------------------------------------------------------
------------
2022/11/10 06:32:45.763075 0x1000000e37c92
f018.985d.3d67 CLIENT_STAGE_TIMEOUT State =
WEBAUTH_REQUIRED, WLAN profile = CWA-TEST2, Policy
profile = flex_vlan4_cwa, AP name = ap3800i-r3-sw2-Gi1-
0-35
Always-on-Tracing
If there is no event for a client-reported issue in the predefined failure
conditions, you can use the Always-on-Tracing feature to perform further
analysis. It enables you to check events that have occurred in the past,
even without having any debugs enabled. With this feature, every control
plane process on the Cisco 9800 WLC is constantly logging at the logging
level of Notice to its dedicated buffer. It allows you to get contextual data
when troubleshooting a specific issue that has occurred without
mandating that the failure condition is reproduced. For example, you can
retrieve the context and actions that caused a client or AP disconnections
to check client roaming patterns or the SSIDs where the client had
connected. Furthermore, it can help you establish trends and isolate
causes if the reported client connectivity problem is specific to certain APs
or client MAC addresses.
You can enable higher logging levels if required, per process or for a group
of processes. Still, higher levels will generate more events and reduce the
total overall time that can be logged for that process. Logs can be viewed
in the console, as in the next example, exported in a file, and so on.
So, when you are facing multiple issues, such as client-related problems,
APs, and mobility, you do not need to use a list of different debug
commands for each scenario since you only need to specify the desired
condition. Then, you must reproduce the problem, generate logs, and
inspect them to resolve it. The following figure illustrates the Radioactive
Trace feature with the Troubleshooting menu in the GUI:
Before you can start the trace, you must specify the required time interval.
It defines how long back you want your collated log file to go and should
cover the period you need for the analysis. Note that MAC address vs IP
address as a condition provides different outputs as different processes
are aware of different identifiers for the same network entity (AP or client
or mobility peer).
You can perform radioactive tracing in the Cisco WLC CLI as well, using
conditional debugging. Similarly, you can focus on specific MAC or IP
address traversing the system. For example, when you troubleshoot the
AP connectivity, you can leverage conditional debugging, which can use
the AP MAC address to get an end-to-end view of the control plane. To
enable conditional debugging via CLI, use this command:
Note
The debug wireless command also allows you to point to an FTP-server and
run even more verbose logging with the keyword internal. However, it is
not recommended to use these commands currently because some issues
are being ironed out.
To redirect the debug output to an external server for offline analysis, use
this command:
To see a much more verbose view of the same debug log levels, use this
command:
Although EPC can be collected easily from the GUI and is sufficient to
capture packets for common use cases, CLI provides more granular
settings for EPC configuration. For example, it can be configured to match
the inner MAC address, which allows you to focus on traffic related to
specific client events when CAPWAP is encapsulated.
The Cisco 9800 Series Wireless Controller also supports the data-path
packet trace feature as part of the Cisco IOS XE platform, which provides
a data plane view of specified traffic. More precisely, it provides a detailed
view of this Cisco IOS XE processing done at the data plane and the
decision made on whether to punt, forward, drop, or consume the packet.
With this feature, you can collect a finite number of packets and include
the conditional debugging ID that can be used to correlate data plane
packet to control plane debugs, timestamp, and feature-specific path-
trace data. Thus, it is not a packet capture feature like EPC. In addition, for
wireless traffic, the data plane packets tracing only parses the outer
CAPWAP header, so conditions like wireless client MAC address do not
yield useful output.
Cisco WCAE is available in two versions, Cloud, where you can upload
your file online, or as a mini-Desktop application. It is fully offline to the
controller. It does not store any data or send any data back. This figure
shows the mini-Desktop application with a report in an Excel file.
When using the Cisco WCAE, consider the following operation and
features requirements:
This tool enables you to perform logical analysis based on log sequence
matching against existing issues. With Cisco 9800 WLC, you can use this
tool to parse through always-on traces and radioactive trace logs. Hence,
you can use these WLC logs more effectively to resolve wireless issues in
your environment. In addition, you can download the parsed file as CSV,
for offline viewing and analysis.
Rogue APs from external parties can also be used to disrupt operations by hijacking
legitimate clients' sessions and performing denial-of-service or man-in-the-middle attacks.
That is, a hacker can use a rogue AP to capture sensitive information, such as usernames and
passwords. The hacker can then transmit a series of Clear to Send (CTS) frames. This action
mimics an AP, informing a particular client to transmit and instructing all the other clients to
wait, which results in legitimate clients being unable to access network resources. Wireless
LAN service providers have a strong interest in banning rogue APs from the air space.
It is important for you to know how to monitor the wireless network for rogue devices,
classify rouge APs, and manage the rogue devices.
Both deployment models support RF detection and are not limited to rogue APs. They can
also capture information upon detection of ad hoc clients and rogue clients (the users of rogue
APs). In monitor mode, the AP is dedicated to scanning the RF channels but does not pass
client data.
A rogue device is an unknown AP or client that managed APs in your network detect as not
belonging to your system. When searching for rogue devices, an AP goes off channel for
50ms to listen for rogue APs, listen for rogue clients, and monitor for noise and channel
interference (the channels to be scanned are configured in the global WLAN network
parameters for IEEE 802.11a/n/ac and IEEE 802.11b/g/n). Information about detected rogue
clients or APs is sent to the Cisco WLC, which gathers the following information:
Cisco WLC then waits to label a rogue client or AP until another AP reports it or until it
completes another cycle.
When the same AP again moves to the same channel to monitor for rogue APs or clients,
noise, and interference and detects the same clients or APs, it lists them as rogues on Cisco
WLC.
Cisco WLC now begins to determine whether a rogue is attached to the local network or is
simply a neighboring AP. In either case, an AP that is not part of the managed local WLAN is
considered a rogue.
In monitor mode, the AP does not carry user traffic but spends all its time scanning channels.
This mode of deployment is most common when a customer does not want to support WLAN
services in a particular area but wants to monitor that area for rogue APs and rogue clients. It
enables you to detect a large number of rogue APs and clients with high sensitivity since it is
used only for monitoring.
Rogue AP Classification
The controller software enables you to create a rule-based classification of the rogue APs, as
the following type:
• Friendly: A rogue AP that matches the friendly access point rule. The rogue state can
be the following:
1. Internal: If the unknown AP poses no threat to WLAN security, you can
manually configure it as Friendly, Internal. An example of this would be the
APs in your lab network.
2. External: If the unknown AP is outside the network and poses no threat to
WLAN security, you can manually configure it as Friendly, External. An
example of this would be the AP in your neighboring coffee shop.
3. Alert: No action is taken other than notifying the management station. The
management station manages the controller and wired networks.
• Malicious: A rogue AP that matches the malicious access point rule. The rogue state
can be the following:
1. Alert: No action is taken other than notifying the management station. The
management station manages the controller and wired networks.
2. Threat: The unknown AP is found to be on the network and poses a threat to
WLAN security.
3. Contained: The unknown AP is contained. If none of the managed APs are
available for containment, the rogue is in a Contained Pending state.
• Custom: A rogue AP that matches the custom rule. The rogue state can
be Alert or Contained.
• Unclassified: A rogue AP that matches no classification rules. The rogue state can
be Alert or Contained.
By default, none of the classification rules are enabled. Therefore, all unknown APs are
categorized as unclassified. The unclassified APs are reclassified when you create a rule,
configure conditions for it, and enable the rule. Whenever you change a rule, it applies to all
APs (friendly, malicious, custom, and unclassified).
Note
You can configure up to 64 rogue classification rules per controller
When the Cisco WLC receives a rogue report from one of its managed APs, it responds as
follows:
• The Cisco WLC verifies whether the unknown AP is in the Friendly MAC address
list. If it is, the controller classifies the AP as Friendly.
• If the unknown AP is not in the Friendly MAC address list, the controller starts
applying rogue classification rules.
• If the rogue AP is manually classified, rogue rules are not applied.
• If the rogue AP matches the configured rules criteria, the controller classifies the
rogue based on the classification type configured for that rule.
• If the rogue AP matches none of the configured rules, the Cisco WLC classifies the
rogue as unclassified.
• The Cisco WLC repeats the previous steps for all rogue APs.
• Suppose Rogue Location Discovery Protocol (RLDP) determines that the rogue AP is
on the same wired network. In that case, the Cisco WLC automatically marks the
rogue state as a Threat and classifies it as Malicious, even if you configured no rules.
You can then manually contain the rogue (unless you have configured RLDP to
contain the rogue automatically), which would change the rogue state to contained. If
the rogue AP is not on the network, the controller marks the rogue state as Alert, and
you can manually contain the rogue.
• If desired, you can manually move the AP to a different classification type and rogue
state.
Before performing any classification, the rogue APs are temporarily marked as Pending.
• Classifying Custom type rogues is tied to rogue rules. Therefore, it is not possible to
manually classify a rogue as Custom. Custom class change can occur only when you
use rogue rules.
• Some SNMP traps are sent for containment by rule and every 30 minutes for rogue
classification change.
• Rogue rules apply on every incoming new rogue report in the controller in the order
of their priority.
• After a rogue satisfies a higher priority rule and is classified, it does not move down
the priority list for the same report.
• The rogue classification rules are re-evaluated at every report received by the
managed APs. Hence, a rogue AP can move from one state to another, if a different
rule matches the last report.
• If a rogue AP is classified as friendly or ignored, all rogue clients associated with it
are not tracked.
• Until the controller discovers all the APs through neighbor reports from APs, the
rogue APs are kept unconfigured for 3 minutes after they are detected. After 3
minutes, the rogue policy is applied to the rogue APs, and the APs are moved to the
Unclassified, Friendly, Malicious, or Custom class. When rogue APs are kept
unconfigured, no rogue policy has yet been applied.
• When a rogue Basic Service Set Identifier (BSSID) is submitted for containment on
the Cisco 9800 WLC, if the controller has enough resources, it will contain. The APs
that detect the contained rogue AP start broadcasting DEAUTH packets, which will
result in the disconnection of the clients connected to that rogue BSSID. However, the
clients will likely repeatedly try to reconnect, and the wireless client's experience will
be badly affected.
Also, in a high RF environment like that of a stadium, though DEAUTH packets are
broadcast, the client does not necessarily receive all of them because of the competing
RF. In this scenario, the client may not be fully disconnected, but its performance can
be impacted badly.
Configuring Rogue Classification Rules
You can configure rogue classification rules in the WLC using the GUI or CLI. To configure
rogue classification rules in the Cisco 9800 WLC GUI navigate
to Configuration > Security > Wireless Protection Policies, and choose the Rogue AP
Rules tab, where you can create rules as indicated in this example.
When creating a rule in the CLI, you must enter the priority for the rule. Then you can
specify the classification that needs to be applied when a rogue AP matches the rule, as well
as apply a condition in the rule that a rogue AP must meet: You can define multiple
conditions and specify whether all or any conditions should be matched.
To create or edit an existing rule in the CLI, as well as specify the classification that needs to
be applied and add a condition in the rule itself, use the following commands.
This example shows how to apply a condition that a rogue AP must meet:
The wireless monitoring option in the Cisco 9800 WLC GUI enables you to observe the
rogues and perform classification manually. Navigate
to Monitoring > Wireless > Rogues to see the options as indicated in the following figure,
where the rogue AP details would be listed. For example, in the Unclassified tab, if there are
unclassified rogues APs in the list, you can select the desired AP and set the status.
To detect and report an ad hoc rogue in the WLC CLI, use this command:
When using the adhoc keyword in the command, you can use the following options:
• Alert: Sets the ad hoc rogue AP to alert mode. If you choose this option, enter the
MAC address for the mac-addr parameter.
• Auto-contain: Sets the automatically containing ad hoc rogue to autocontain mode.
• Auto-contain: Sets the automatically containing ad hoc rogue to autocontain mode.
• Contain: Sets the containing ad hoc rogue AP to contain mode. If you choose this
option, enter the MAC address for the mac-addr parameter and containment level for
the containment-level parameter. The valid range for containment-level is from
1 to 4.
• External: Sets the ad hoc rogue AP as external. If you choose this option, enter the
MAC address for the mac-addr parameter.
• Internal: Sets the ad hoc rogue AP as internal. If you choose this option, enter the
MAC address for the mac-addr parameter.
For example, in the global configuration mode, use the wireless wps rogue adhoc alert
74a0.2f45.c520 command to set the rogue AP with the specified MAC to alert mode.
You can also manually classify a rogue AP using the following command.
You can also configure rogue clients while using the following command.
For example, with the wireless wps rogue client contain 74a0.2f45.c520
2 command, you can report a rogue client.
In the Rogues section of the Security tab of the AP Join Profile, you can use the following:
You can adjust the rogue detection settings in the WLC CLI as well while configuring the
selected AP Join Profile. In addition, you can use different show commands to verify the
rogue detection, as in the example below, verify rogue auto-containment information, display
rogue classification rules, inspect rogue statistics, and so on. The following command
provides various verification options that can be beneficial during rogue devices detection.
The following example shows information for rogue AP detection, which includes
authentication errors. As a result, this AP is classified as Malicious, and its state is set
to Threat.
Number of clients : 0
Reported By
AP Name : AP38ED.18CE.45E0
MAC Address : 38ed.18cf.83e0
Detecting slot ID : 0
Radio Type : dot11g, dot11n - 2.4
GHz
SSID : rogueA
Channel : 6 (From DS)
Channel Width : 20 MHz
RSSI : -33 dBm
SNR : 52 dB
ShortPreamble : Disabled
Security Policy : WPA2/WPA/FT
Last reported by this AP : 01/08/2020 08:02:53
Authentication Failure Count : 237
You can adjust the existing parameters if needed, such as the rogue AP expiration timeout, as
well as you can enable additional features in the policy, such as to validate rogue clients and
APs against the AAA server, and so on.
Early in the process of drafting the 802.11 standards, a great concern emerged that the
proliferation of Wi-Fi networks would create interference for licensed users that operate in
the same frequency bands. As a result of this concern, Wi-Fi was very “polite” and yielded
the band to almost anything else it found operating there. Twenty years later, many consumer
devices share the industrial, scientific, and medical (ISM) bands with Wi-Fi. Although these
devices operate under the same output power restrictions as Wi-Fi devices, they are not
obligated to yield the band for Wi-Fi traffic, and most do not.
This situation creates a problem for normal Wi-Fi operations because a Wi-Fi modem can
classify energy only as follows:
The symptoms of interference may be different, but generally, they can be in the following
categories:
Cisco CleanAir
The Cisco CleanAir technology is a system-wide feature streamlining operations and
improving wireless performance by providing complete visibility into the spectrum. Suppose
an interference source is strong enough to jam a Wi-Fi channel completely. In that case,
the system will change channels within seconds to avoid interference, resuming client activity
on another channel outside the affected area.
Cisco CleanAir technology mitigates the impact of wireless interference in all Wi-Fi bands,
including 2.4 GHz and 5 GHz. Cisco CleanAir Pro extends the solution to the 6 GHz band
used by Wi-Fi 6E.
The Cisco CleanAir system consists of CleanAir-enabled APs and the Cisco Catalyst 9800
Series WLC.
• APs collect information about all the devices that operate in the industrial, scientific,
and medical (industrial, scientific, and medical [ISM]) bands, identify and evaluate
the information as a potential interference source, and forward it to the controller.
• The controller controls and configures CleanAir-capable APs and collects and
processes spectrum data. The controller provides local user interfaces (GUI and CLI)
to configure basic CleanAir features and services and display current spectrum
information. The controller also detects, merges, and mitigates interference devices
using RRM Transmit Power Control (TPC) and Dynamic Channel Assignment
(DCA)
For every device operating in the unlicensed band, Cisco CleanAir provides information
about what it is, how it is impacting your wireless network, and what actions to take. Cisco
CleanAir monitors the full channel bandwidth capability of a Cisco CleanAir-capable access
point (20 MHz—160 MHz channels).
To enable Cisco CleanAir on the Cisco Catalyst 9800 WLC in the GUI, navigate
to Configuration > Radio Configurations > CleanAir page and check the Enable
CleanAir check box for the desired frequency band.
In addition, you can configure Interference reporting for a 5-GHz and 2.4-GHz device by
choosing the interference types and adding them to the Interference Types to detect section
in the 5-GHz and 2.4-GHz bands. Hence, you can detect time division duplex (TDD)
transmitters, jammers, continuous transmitters, DECT-like phones, video cameras (used for
analog video cameras), devices that use spectrally inverted Wi-Fi signals, devices that uses
nonstandard Wi-Fi channels, and so on.
Spectrum Intelligence
Maintaining a healthy wireless environment requires knowledge of what is in the air.
Spectrum intelligence is a core technology that is designed for proactively managing the
challenges of a shared wireless spectrum by scanning for non-Wi-Fi radio interference on
2.4-GHz and 5-GHz bands.
Essentially, spectrum intelligence is data about RF spectrum activity that is derived from
advanced interference identification algorithms. Spectrum intelligence provides visibility of
all the users of the shared spectrum: both Wi-Fi devices and non-Wi-Fi interferers. For every
device that operates in the unlicensed band, spectrum intelligence tells you what the device is,
where the device is, and how it is affecting the Wi-Fi network.
The first step in effectively managing the environment is to know what is in the wireless
spectrum:
Spectrum management is the active use of spectrum intelligence data to improve performance
and lower the operational costs of Wi-Fi networks. Information about the severity and
duration of interference can be used to calculate its impact on the network and to troubleshoot
problems. This information can also be stored for historical analysis and trending.
Note
Keep in mind that Cisco CleanAir provides spectrum management.
Combined with contextual data like physical location and system-wide correlation, spectrum
management is a powerful, proactive tool that increases WLAN reliability, performance, and
security.
To configure Spectrum Intelligence on the Catalyst 9800 WLC using GUI, navigate
to Configuration > Radio Configurations > CleanAir page and check the Enable SI check
box for the desired frequency band.
In the CleanAir Interference Devices, you can view the following information:
In the same menu, you can view similar information for the spectrum intelligence
interference devices.
With Cisco CleanAir, all energy within the spectrum that is definitely not Wi-Fi is accounted
for and classified. The system interprets energy that is 802.11-modulated and classifies
energy that is coming from co-channels and adjacent channel sources.
For each classified device, the system calculates a severity index, a positive integer from 0 to
100, with 100 being the most severe. The index is then subtracted from the air quality scale (0
to 100, with 100 being good) to generate the actual Air Quality Index (AQI) for a channel,
AP, floor, building, or campus. Air quality (AQ) is a rolling average that indicates the impact
of all classified interference devices against a theoretical perfect spectrum. For example, an
AQI of 10 is wrong, while an AQI > 85 is good.
In the CleanAir Statistics page, you can access the Air Quality Report for each radio band,
as indicated in this figure.
In the Air Quality Report, you can view the following information:
• AP Name: The name of the AP that reported the worst air quality.
• AP MAC: The MAC of the AP that reported the worst air quality.
• Channel: The radio channel where the air quality is monitored.
• Average AQ: The average air quality for this radio channel.
• Minimum AQ: The minimum air quality for this radio channel.
• Number of Interferers: The number of interferers detected by the radios.
• Detected Time: The time when the interferer was detected.
Within the Clear Air statistics page, you can also view the Worst Air Quality Report, which
provides the worst air quality information for the radio band.
The AQI Alarm Threshold enables you to specify a value between 1 and 100 for the
threshold at which you want the air quality alarm to be triggered. Hence, when the air quality
falls below the threshold level, the alarm is triggered, and you will get a trap notification that
there is an unusual level of interference in your wireless environment.