Troubleshoot High CPU Utilization Due to SNMP Process

WLC#show snmp stats hosts
Request Count    Last Timestamp  Address
77888844          00:00:00 ago   10.10.10.120
330242            00:00:08 ago   10.10.10.150
27930314          00:00:09 ago   10.10.10.130
839999            00:00:36 ago   10.10.10.170
6754377           19:45:34 ago   10.10.10.157
722               22:00:20 ago   10.10.10.11

This command provides a list of SNMP servers along with their request counts and the last timestamp of their polling activity.

You can see there are multiple different server that are polling the 9800 WLC. If you look at the complete logs data collected during the last 10 mins you can gauge their polling frequency as well.

Now you can go to each server and see how frequently the offending OID is being polled. In this example case, the OID is being polled every 30 seconds, which is significantly more frequent than necessary. Since the WLC receives RF/RRM data every 180 seconds, polling the OID every 30 seconds results in unnecessary processing and contributes to high CPU utilization.

Once the offending OID and the server have been identified, we can try multiple different solution to reduce the load on the WLC.

1. Reduce the polling frequency on the SNMP server.
2. If the OID is not needed for operation usage, disable polling of that OID from the SNMP server.
3. If you do not have control over the SNMP server, you can use SNMP view to block the offending OID.

SNMP View configuration

Define a new view that excludes the OID you want to block. For example, you want to block the OID 1.3.6.1.4.1.14179.2.2.15.1.21, create a new view and attach the OID to the view.

snmp-server view blockOIDView 1.3.6.1.4.1.14179.2.2.15.1.21 excluded  <-- This is the OID of bsnAPIfDBNoisePower
snmp-server community TAC view blockOIDView RO <-- This command assigns the blockOIDView to the community myCommunity with read-only (RO) access.
snmp-server group TAC v3 priv read blockOIDView <-- This command assigns the blockOIDView to the group myGroup with the priv security level for SNMPv3.

Troubleshooting Tip

• Baseline CPU Usage: Document the normal CPU utilization levels when the SNMP process is not causing high usage.
• SNMP Configuration: Review the current SNMP configuration settings, including community strings, version (v2c or v3), and access lists.
• SNMP Best Practice: Use the 9800 WLC best practise document and match the suggested configuration for SNMP as close as possible.

C9800(config)#snmp-server subagent cache
C9800(config)#snmp-server subagent cache timeout ?
  <1-100>  cache timeout interval (default 60 seconds)

• Frequency of SNMP Polling: Determine how often the WLC is being polled by SNMP queries, as a high frequency could contribute to increased CPU load.
• Network Topology and SNMP Managers: Understand the network setup and identify all SNMP managers that are interacting with the WLC.
• System Uptime: Check the time elapsed since the last reboot to see if there is a correlation between uptime and CPU utilization.
• Recent Changes: Note any recent changes to the WLC configuration or network that can have coincided with the onset of high CPU utilization.
• With 9800 WLC, the focus has been put on telemetry. Telemetry works in a "push" model where WLC sends out relevant information to the server without the need to be queried. If your SNMP queries are consuming WLC CPU cycles and causing operation issues, it would be better to move to Telemetry.

Conclusion

By methodically analyzing CPU utilization data and correlating it with SNMP polling activities, you can troubleshoot and resolve high CPU utilization issues caused by SNMP processes on the Cisco 9800 WLC. Post-implementation monitoring is essential to confirm the success of the troubleshooting efforts and to maintain optimal network performance.

Related Information

• Monitor Catalyst 9800 WLC via SNMP with OIDs
• Manage Catalyst 9800 Wireless Controller Series with Prime Infrastructure with SNMP V2 and V3 and NetCONF
• Secure Your Simple Network Management Protocol
• Cisco Catalyst 9800 Series Configuration Best Practices