Contents

Introduction

Methodology

Check for Output Drops

Determine if the Drops are Unicast or Multicast

Determine Which Output Buffer is Used

Check the Active Buffer Monitoring

Counters Actively Increment

Brief Output

Detailed Output

Generate a Log When a Threshold Is Crossed

Notable Cisco Bug IDs

Frequently Asked Questions

Appendix - Feature Information

Buffer Management

Scheduling

Multicast Slow-Receiver

Active Buffer Monitoring

Hardware Implementation

Software Implementation

Introduction

This document describes commands used in order to troubleshoot the type of traffic dropped on the Nexus 3500 platform and the output buffer (OB) in which this traffic is dropped.

Methodology

1. Check for Output Drops

2. Determine if the Drops are Unicast or Multicast

3. Determine Which Output Buffer is Used

4. Check the Active Buffer Monitoring

Check for Output Drops

Check the physical interface statistics in order to determine if the traffic is dropped in the egress direction. Determine if the "output discard" counter in the TX direction increments and/or is non-zero.

Nexus3548# show interfce Eth1/7

Ethernet1/7 is up

Dedicated Interface

Hardware: 100/1000/10000 Ethernet, address: a44c.116a.913c (bia a44c.116a.91ee)

Description: Unicast Only

Internet Address is 1.2.1.13/30

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

reliability 255/255, txload 35/255, rxload 1/255

Encapsulation ARPA

full-duplex, 1000 Mb/s, media type is 1G

Beacon is turned off

Input flow-control is off, output flow-control is off

Rate mode is dedicated

Switchport monitor is off

EtherType is 0x8100

Last link flapped 00:03:48

Last clearing of "show interface" counters 00:03:55

1 interface resets

30 seconds input rate 200 bits/sec, 0 packets/sec

30 seconds output rate 0 bits/sec, 0 packets/sec

Load-Interval #2: 5 minute (300 seconds)

input rate 40 bps, 0 pps; output rate 139.46 Mbps, 136.16 Kpps

RX

1 unicast packets  118 multicast packets  0 broadcast packets

119 input packets  9830 bytes

0 jumbo packets  0 storm suppression bytes

0 runts  0 giants  0 CRC  0 no buffer

0 input error  0 short frame  0 overrun   0 underrun  0 ignored

0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

0 input with dribble  0 input discard

0 Rx pause

TX

23605277 unicast packets  0 multicast packets  0 broadcast packets

23605277 output packets  3038908385 bytes

0 jumbo packets

0 output errors  0 collision  0 deferred  0 late collision

0 lost carrier  0 no carrier  0 babble 11712542 output discard

0 Tx pause

Determine if the Drops are Unicast or Multicast

Once it is determined that the interface drops the traffic, enter the show queuing interface <x/y> command in order to find out if the traffic dropped is multicast or unicast. In releases earlier than 6.0(2)A3(1), the output looks like:

Nexus3548# show queuing interface Eth1/7

Ethernet1/7 queuing information:

TX Queuing

qos-group  sched-type  oper-bandwidth

0       WRR            100

RX Queuing

Multicast statistics:

Mcast pkts dropped                      : 0

Unicast statistics:

qos-group 0

HW MTU: 1500 (1500 configured)

drop-type: drop, xon: 0, xoff: 0

Statistics:

Ucast pkts dropped                      : 11712542

In Release 6.0(2)A3(1) and later, the output looks like:

Nexus3548# show queuing interface Eth1/7

Ethernet1/7 queuing information:

qos-group  sched-type  oper-bandwidth

0       WRR            100

Multicast statistics:

Mcast pkts dropped                      : 0

Unicast statistics:

qos-group 0

HW MTU: 1500 (1500 configured)

drop-type: drop, xon: 0, xoff: 0

Statistics:

Ucast pkts dropped                      : 11712542

Note: If the multicast slow receiver is configured for the port, see for feature information, drops are not tracked with the show queuing interface Eth<x/y> command due to a hardware limitation. See Cisco bug ID CSCuj21006.

Determine Which Output Buffer is Used

In Nexus 3500, there are three buffer pools used in the egress direction. The output of the show hardware internal mtc-usd info port-mapping command provides the mapping information.

Nexus3548# show hardware internal mtc-usd info port-mapping

OB Ports to Front Ports:

========= OB0 =========    ========= OB1 =========    ========= OB2 =========

45 47 21 23 09 11 33 35    17 19 05 07 41 43 29 31    13 15 37 39 25 27 01 03

46 48 22 24 10 12 34 36    18 20 06 08 42 44 30 32    14 16 38 40 26 28 02 04

Front Ports to OB Ports:

=OB2= =OB1= =OB0= =OB2=    =OB1= =OB0= =OB2= =OB1=    =OB0= =OB2= =OB1= =OB0=

12 14 04 06 08 10 00 02    00 02 04 06 08 10 12 14    12 14 04 06 08 10 00 02

13 15 05 07 09 11 01 03    01 03 05 07 09 11 13 15    13 15 05 07 09 11 01 03

Front port numbering (i.e. "01" here is e1/1):

=OB2= =OB1= =OB0= =OB2=    =OB1= =OB0= =OB2= =OB1=    =OB0= =OB2= =OB1= =OB0=

01 03 05 07 09 11 13 15    17 19 21 23 25 27 29 31    33 35 37 39 41 43 45 47

02 04 06 08 10 12 14 16    18 20 22 24 26 28 30 32    34 36 38 40 42 44 46 48

Note: Text in Red font is _not_ CLI output, it's purely to help those reading

the document faster match the actual front port instead of having to manually

count up.

The first part of the results indicate that the OB pool 0 is used by front-ports such as 45, 46, 47, 48, and so on and OB1 is used by front-ports 17, 18, and so on.

The second part of the results indicate that Eth1/1 is mapped to OB2 port 12, Eth1/2 is mapped to OB2 port 13, and so on.

The port in discussion, Eth1/7, is mapped to OB1.

See the Buffer Management section in this document for further information.

Check the Active Buffer Monitoring

See the Cisco Nexus 3548 Active Buffer Monitoring whitepaper and the section in this document for further information about this feature.

Counters Actively Increment

If the output discards actively increment, enable Active Buffer Monitoring (ABM) with this command. Note that the command allows you to monitor unicast or multicast, but not both. Also, it lets you configure the sampling interval and threshold values.

hardware profile buffer monitor [unicast|multicast] {[sampling <interval>] |

[threshold <Kbytes>]}

Brief Output

Once the ABM is enabled, you can view the results with this command.

Nexus3500# show hardware profile buffer monitor interface e1/7 brief

Brief CLI issued at: 09/30/2013 19:43:50

Maximum buffer utilization detected

1sec     5sec    60sec     5min      1hr

------   ------   ------   ------   ------

Ethernet1/7       5376KB   5376KB   5376KB      N/A      N/A

These results indicate that 5.376 MB out of 6 MB of the OB1 buffer has been used by unicast traffic that left Eth1/7 for the past 60 seconds.

Detailed Output

Nexus3500# show hardware profile buffer monitor interface Eth1/7 detail

Detail CLI issued at: 09/30/2013 19:47:01

Legend -

384KB  - between   1 and 384KB of shared buffer consumed by port

768KB  - between 385 and 768KB of shared buffer consumed by port

307us  - estimated max time to drain the buffer at 10Gbps

Active Buffer Monitoring for Ethernet1/7 is: Active

KBytes                 384  768 1152 1536 1920 2304 2688 3072 3456 3840 4224 4608 4992 5376 5760 6144

us @ 10Gbps            307  614  921 1228 1535 1842 2149 2456 2763 3070 3377 3684 3991 4298 4605 4912

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

09/30/2013 19:47:01      0    0    0    0    0    0    0    0    0    0    0    0    0  250    0    0

09/30/2013 19:47:00      0    0    0    0    0    0    0    0    0    0    0    0    0  252    0    0

09/30/2013 19:46:59      0    0    0    0    0    0    0    0    0    0    0    0    0  253    0    0

09/30/2013 19:46:58      0    0    0    0    0    0    0    0    0    0    0    0    0  250    0    0

09/30/2013 19:46:57      0    0    0    0    0    0    0    0    0    0    0    0    0  250    0    0

09/30/2013 19:46:56      0    0    0    0    0    0    0    0    0    0    0    0    0  250    0    0

09/30/2013 19:46:55      0    0    0    0    0    0    0    0    0    0    0    0    0  251    0    0

09/30/2013 19:46:54      0    0    0    0    0    0    0    0    0    0    0    0    0  251    0    0

09/30/2013 19:46:53      0    0    0    0    0    0    0    0    0    0    0    0    0  250    0    0

09/30/2013 19:46:52      0    0    0    0    0    0    0    0    0    0    0    0    0  253    0    0

09/30/2013 19:46:51      0    0    0    0    0    0    0    0    0    0    0    0    0  249    0    0

...

Information in each row is logged at a second interval. Each column represents the buffer usage. As mentioned in the command results, if there is a non-zero value reported for column "384", it means that buffer usage was between 0-384 KBytes when the ABM polled the OB usage. The non-zero number is the number of times the usage was reported.

These results indicate that OB1 averaged 5.376 MB of usage between 249 - 253 times for each second in the last 10 seconds for Eth1/7. It takes 4298 microseconds (us) in order to clear the buffer of this traffic.

Generate a Log When a Threshold Is Crossed

If the drop counter and buffer usage increments periodically, then it is possible to set a threshold and generate a log message when the threshold is crossed.

logging level mtc-usd 5

hardware profile buffer monitor unicast sampling 10 threshold 4608

The command is set to monitor unicast traffic at a 10 nanoseconds interval and when it goes above 75% of the buffer it generates a log.

You can also create a scheduler in order to collect ABM statistics and interface counter output every hour and append it to bootflash files. This example is for multicast traffic:

hardware profile buffer monitor multicast

feature scheduler

scheduler job name ABM

show hardware profile buffer monitor detail >> ABMDetail.txt

show clock >> ABMBrief.txt

show hardware profile buffer monitor brief >> ABMBrief.txt

show clock >> InterfaceCounters.txt

show interface counters errors >> InterfaceCounters.txt

scheduler schedule name ABM

time start now repeat 1:0

job name ABM

Notable Cisco Bug IDs

- Cisco bug ID CSCum21350: Rapid port flaps cause all ports in same QoS buffer to drop all TX Multicast/Broadcast traffic. This is fixed in Release 6.0(2)A1(1d) and later.

- Cisco bug ID CSCuq96923: The multicast buffer block is stuck, which results in egress multicast/broadcast drops. This issue is still under investigation.

- Cisco bug ID CSCva20344: Nexus 3500 Buffer block / lockup - no TX multicast or broadcast. Unreproducible issue, potentially got fixed in Releases 6.0(2)U6(7), 6.0(2)A6(8), and 6.0(2)A8(3).

- Cisco bug ID CSCvi93997:  Cisco Nexus 3500 switches output buffer block stuck. This is fixed in Releases 7.0(3)I7(8) and 9.3(3).

Frequently Asked Questions

Does ABM impact performance or latency?

No, this feature does not impact latency or performance of the device.

What is the impact of the lower ABM hardware polling interval?

By default, the hardware polling interval is 4 milliseconds. You can configure this value as low as 10 nanoseconds. There is no performance or latency impact because of the lower hardware polling interval. The default hardware polling of 4 milliseconds is selected in order to make sure you do not overflow the histogram counters before the software polls every one second. If you lower the hardware polling interval then it might saturate the hardware counters at 255 samples. The device cannot handle a software polling lower than one second, in order to match the lower hardware polling due to CPU and memory restrictions. The whitepaper has the example of the lower hardware polling interval and its use case.

Appendix - Feature Information

Buffer Management

- 18 MB packet buffer shared by three OB blocks:

- ~4 MB Reserved: Size based on configured Maximum Transmission Unit (MTU) (Per port sum of 2 x MTU Size x # of enabled QoS groups)

- ~14 MB Shared: Remainder of total buffer

- ~767 KB of OB: 0 for CPU destined packets

- 6 MB for each OB is shared by a set of 16 ports (show hardware internal mtc-usd info port-mapping command)

Scheduling

Three-layer scheduling:

- Unicast and multicast

- Traffic classes of the same scheduling scheme

- Traffic classes across the scheme

Multicast Slow-Receiver

In this diagram:

- Sustained congestion is introduced on 1 G Eth1/40.

- Other multicast receivers (Eth1/1 - 3) on the buffer block are impacted due to the multicast scheduling behavior.  Receivers on other buffer blocks remain unaffected.

- "Multicast slow-receiver" can be applied to e1/40 in order to avoid traffic loss on non-congested ports.

- "Multicast slow-receiver" drains the multicast at a 10 G rate on Eth1/40. Drops are still expected to occur on the congested port.

- Configured with the hardware profile multicast slow-receiver port <x> command.

Active Buffer Monitoring

See the Cisco Nexus 3548 Active Buffer Monitoring whitepaper for an overview of this feature.

Hardware Implementation

- ASIC has 18 buckets and each bucket corresponds to a range of buffer utilization (for example, 0-384KB, 385-768KB, and so on).

- ASIC polls the buffer utilization for all ports every 4 milliseconds (default).  This ASIC polling interval is configurable as low as 10 nanoseconds.

- Based on the buffer utilization for each hardware polling interval, the bucket counter for the corresponding range is incremented. That is, if port 25 consumes 500 KB of buffer, the bucket #2 (385-768KB) counter is incremented.

- This buffer utilization counter is maintained for each interface in histogram format.

- Each bucket is represented with 8 bits, so the counter maxes out at 255 and it is reset once the software reads the data.

Software Implementation

- Every one second, the software polls ASIC in order to download and clear all histogram counters.

- These histogram counters are maintained in the memory for 60 minutes with one second granularity.

- The software also makes sure it copies the buffer histogram to the bootflash every hour, which can be copied to the analyzer for further analysis.

- Effectively, this maintains two hours worth of buffer histogram data for all ports, the latest one hour in the memory and second hour in the bootflash.