Migrate to Dual Supervisors for Nexus 9800 Modular Switch

Introduction

This document describes how to migrate to Dual Supervisors for the Nexus 9800 modular switches.

Prerequisites

Requirements

Console access to both Supervisors is necessary.

Cisco recommends that you understand the basics of Upgrading Nexus 9000 switches.

For more information, refer to the documentation here:

Cisco Nexus 9000 Series NX-OS Software Upgrade and Downgrade Guide, Release 10.4(x)

Cisco recommends that you understand the basics of High Availability for Nexus 9000 switches.

For more information, refer to the documentation here:

Cisco Nexus 9000 Series NX-OS High Availability and Redundancy Guide, Release 10.4(x)

Components Used

The information in this document is based on Nexus 9808 running NX-OS software release 10.4(3)F with two N9K-C9800-SUP-A Supervisor Modules.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The Cisco Nexus 9000 Series switches are protected from the impact of hardware or software failures by redundant hardware components.

Dual supervisors provide redundancy for the control and management plane. The two supervisors operate in an active/standby capacity in which only one of the supervisor modules is active at any given time, while the other acts as a standby backup. The two supervisors constantly synchronize the state and configuration to provide a seamless and stateful switchover of most services if the active supervisor module fails.

Support for Dual Supervisors was introduced for the Nexus 9808 switches in 10.4(1)F and the Nexus 9804 switches in 10.4(2)F respectively.

Early platform deployments can contain a single supervisor initially and transition to dual supervisors.

It is important to understand how to correctly migrate to Dual Supervisors on this platform.

The example outputs referenced throughout this document are from a Nexus 9808 running NX-OS software release 10.4(3)F.
This content is also applicable to a Nexus 9804.

In this document the terms Secondary Supervisor and Standby Supervisor are used.

Secondary Supervisor refers to the second supervisor physically installed into the chassis but has not necessarily synced with the Active Supervisor.

Standby Supervisor refers to the second supervisor physically installed into the chassis and has synced with the Active Supervisor in the HA-Standby state.

Before You Get Started

Firstly, upgrade your Nexus 9800 switch with a single supervisor to a NX-OS software release version with dual supervisor support as mentioned in the previous section.

For deploying features only supported in 10.4(x), Cisco recommends upgrading to the latest available release of 10.4(x) referenced by the Recommended Cisco NX-OS Releases for Cisco Nexus 9000 Series Switches.

Insert Secondary Supervisor

Proceed with inserting Secondary Supervisor into supervisor slot 28 on the Nexus 9800.
Insert console cable into Secondary Supervisor and open a terminal session to monitor the boot process.

Secondary With 10.4(x) Release

A Secondary Supervisor with a 10.4(x) release pre-installed will recognize itself as the Standby Supervisor and slot 27 supervisor as the Active Supervisor.
The Secondary Supervisor will begin to sync with the Active Supervisor starting with their boot images.

Cisco N9800(R) Series BIOS Ver 1.11 Primary
Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz
Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU4
Chipguard Init
TAM Library Version: 3.4.26
Aikido Version: 5.0.13
Aikido Chip Serial Number:
006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000
Aikido FPGA Version: 0x120
Aikido chip authentication successful
Aikido KEY Format: UEFI
Cisco PK Key
Cisco Key Key
Cisco DBX Key
Cisco OS Rel Key
Cisco OS Dev Key

Board type 5
IOFPGA @ 0xb2400000
SLOT_ID @ 0x1c

NXOS Grub
disable watchdog
Filesystem type unknown, partition type 0x83
check_bootmode: grub: Continue grub
Trying to read config file /boot/grub/menu.lst.local from (hd0,4)
Filesystem type is ext2fs, partition type 0x83

Booting bootflash:/nxos64-cs.10.4.3.F.bin ...
Booting bootflash:/nxos64-cs.10.4.3.F.bin                  <<<
Trying diskboot
Filesystem type is ext2fs, partition type 0x83
Image valid


Image Signature verification was Successful.

Boot Time: 8/5/2024 17:41:4
mount: overlay mounted on /newroot/usr.
Installing klm_card_index
done
Setting nativeboot
Linking n9k mando SSD partition...
creating flash devices BOOT_DEV= sda
INIT: version 2.88 booting
Installing ata_piix module ... done.
Installing kheaders module ... done.
Unsquashing rootfs ...
Total size needed in bootflash is 158780
check bootflash : OK
Total size needed in bootflash is 55644
check bootflash : OK
Enabling 8250 serial driver spurious INTs workaround
Installing isan procfs ... done.
is_lxc: is_titan_eor: is_stby: suffix: klm_ftrace: /isanboot/lib/modules/klm_ftrace.o
Installing ftrace in non-lxc mode done
Installing SSE module with card index 21213 ... done.
Creating SSE device node 243 ... done.
Executing /etc/rc.d/rcS.d/S06exablaze start
Executing /etc/rc.d/rcS.d/S06exablaze start
MKR mando set cpa libs
MANDO RP CPA setup done
Loading I2C driver ... done.
Installing CCTRL driver for card_type 132 without NEED_GE[ 16.676681] ENABLING TOP LEVEL RP INTERRUPT
M ... [ 16.682862] 28 Stdby waiting for 27 to become active           <<<
[ 18.689037] Now Continuing..
done.

Same Release Image

If the boot image for both supervisors is the same, the Secondary Supervisor will complete boot process and move to establish High Availability standby state (ha-standby) with the Active Supervisor.

User Access Verification
N9K-C9808(standby) login: 2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 16.676681] ENABLING TOP LEVEL RP INTERRUPT - kernel
2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 16.682862] 28 Stdby waiting for 27 to become active - kernel
2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 18.689037] Now Continuing.. - kernel
2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 53.624182] Initializing panic NMI - kernel
2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 53.656637] fpga_setup_probe - kernel
2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 85.571069] Initializing delayed work - kernel
2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 85.575323] Finished INIT function for lfm - kernel
2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 94.522919] MANDO RDN : register_rdn_int - kernel
2024 Aug 5 17:43:17 stark %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [ 94.527406] register_rdn_int : Sent First Heartbeat - kernel
2024 Aug 5 17:43:18 stark %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%USBHSD-2-MOUNT>> logflash: online - usbhsd
2024 Aug 5 17:43:43 stark %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Installing thirdparty RPMs - /thirdparty_install.py
2024 Aug 5 17:43:46 stark %$ VDC-1 %$ %USER-2-SYSTEM_MSG: ssnmgr_app_init called on ssnmgr up - aclmgr
2024 Aug 5 17:43:47 stark %$ VDC-1 %$ %USER-1-SYSTEM_MSG: VP aclqos tah stats get - pltfm_config
2024 Aug 5 17:44:11 stark %$ VDC-1 %$ %USER-1-SYSTEM_MSG: ASSERT@../utils/usd/libusd/usd_drv.c:3332 - s1hal_supe
2024 Aug 5 17:44:16 stark %$ VDC-1 %$ %USER-1-SYSTEM_MSG: ASSERT@../utils/usd/libusd/usd_drv.c:3332 - s1hal_supe (message repeated 1 time)
2024 Aug 5 17:44:16 stark %$ VDC-1 %$ %USER-0-SYSTEM_MSG: end of default policer - copp
2024 Aug 5 17:44:36 stark %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary
2024 Aug 5 17:44:36 stark %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: TMFPGA booted from Primary

N9K-C9808# show module
Mod Ports Module-Type Model Status
--- ----- ------------------------------------------------ --------------------- --------
27 0 Supervisor Module N9K-C9800-SUP-A active *
28 0 Supervisor Module N9K-C9800-SUP-A ha-standby <<<

Different Release Images

If the boot image for both supervisors is different 10.4(x) releases, the Secondary Supervisor will be rebooted to force NetBoot and download the Active Supervisor's boot image to the Secondary Supervisor.

In the this example, our Active Supervisor is running 10.4(3)F but the Secondary Supervisor booted to 10.4(1)F.

Cisco N9800(R) Series BIOS Ver 1.11 Primary
Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz
Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU4
Chipguard Init
TAM Library Version: 3.4.26
Aikido Version: 5.0.13
Aikido Chip Serial Number:
006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000
Aikido FPGA Version: 0x120
Aikido chip authentication successful
Aikido KEY Format: UEFI
Cisco PK Key
Cisco Key Key
Cisco DBX Key
Cisco OS Rel Key
Cisco OS Dev Key

Board type 5
IOFPGA @ 0xb2400000
SLOT_ID @ 0x1c

NXOS Grub
disable watchdog
Filesystem type unknown, partition type 0x83
check_bootmode: grub: Continue grub
Trying to read config file /boot/grub/menu.lst.local from (hd0,4)
Filesystem type is ext2fs, partition type 0x83

Booting bootflash:/nxos64-cs.10.4.1.F.bin ...
Booting bootflash:/nxos64-cs.10.4.1.F.bin     <<<
Trying diskboot
Filesystem type is ext2fs, partition type 0x83
Image valid

<...>

[ 93.294604] MANDO RDN : register_rdn_int
[ 93.299095] register_rdn_int : Sent First Heartbeat

< Reboots into NetBoot process >

Cisco N9800(R) Series BIOS Ver 1.11 Primary
Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz
Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU4
Chipguard Init
TAM Library Version: 3.4.26
Aikido Version: 5.0.13
Aikido Chip Serial Number:
006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000
Aikido FPGA Version: 0x120
Aikido chip authentication successful
Aikido KEY Format: UEFI
Cisco PK Key
Cisco Key Key
Cisco DBX Key
Cisco OS Rel Key
Cisco OS Dev Key

Board type 5
IOFPGA @ 0xb2400000
SLOT_ID @ 0x1c

NXOS Grub
disable watchdog
Finding driver for NIC vendor 8086 Device 15ab
i210 devide id = 1537
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
x check driver for NIC vendor 8086 Device 15ab
inside driver for NIC vendor 8086 Device 15ab
pci_base_addres 0= b2c0000c ioaddr= 0
mask issue
Found the device 8086:15ab at ioaddr 1000, membase b2c00000 at 4:0
Probing...
intelx probe

eobc MAC address 00000000:00000000:00000000:0000001c:00000000:00000000

board_type=5
network is ready.
bp_op = 1
bp_htype = 1
bp_hlen = 6
bp_xid = -160883712
bp_hwaddr = 00:00:00:1C:00:00
bp_hops = 0
bp_secs = 2132726648
*retry 1 currticks 27126 timeout = 720
*retry 2 currticks 27684 timeout = 720
bp_op = 1
bp_htype = 1
bp_hlen = 6
bp_xid = -160883712
bp_hwaddr = 00:00:00:1C:00:00
bp_hops = 0
bp_secs = 2132726648
errnum = 0
reqretry = 0
*2 retry 2 currticks 27684 timeout = 720
Filename: /img-sync/curr-ks.img

board_type=5

board_type=5
Address: 127.1.1.28
Netmask: 255.255.0.0
Server: 127.1.1.27
Gateway: 0.0.0.0
Filesystem type is tftp, using whole disk
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Image valid

Image Signature verification was Successful.

Once the download has completed, the Secondary Supervisor will boot to the new image then move to establish ha-standby state with the Active Supervisor.

N9K-C9808# show module
Mod Ports Module-Type Model Status
--- ----- ------------------------------------------------ --------------------- --------
27 0 Supervisor Module N9K-C9800-SUP-A active *
28 0 Supervisor Module N9K-C9800-SUP-A ha-standby

Mod Sw Hw Slot
--- ----------------------- ------ ----
27 10.4(3) 1.0 SUP1
28 10.4(3) 1.0 SUP2

Secondary With 10.3(x) Release

For a Secondary Supervisor with a 10.3(x) release pre-installed, one of the listed events can happen after insert:
1. Boot loop.
2. Boot as an independent supervisor.
3. Boot into loader > prompt

If the Secondary Supervisor is in the state of event 1 or 2, then you must break it into loader > prompt by typing Ctrl-C at "NXOS Grub" during the bootup process.
Cisco N9800(R) Series BIOS Ver 1.11 Primary
Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz
Board Type 0x220 PID N9K-C9808 Serial FOX2633PCKJ
Chipguard Init
TAM Library Version: 3.4.26
Aikido Version: 5.0.13
Aikido Chip Serial Number:
3E772D0F6E8FC7406CEC11BBA5B2A84C00000000000000000000000000000000
Aikido FPGA Version: 0x11B
Aikido chip authentication successful
Aikido KEY Format: UEFI
Cisco PK Key
Cisco Key Key
Cisco DBX Key
Cisco OS Rel Key
Cisco OS Dev Key

Board type 5
IOFPGA @ 0xb2400000
SLOT_ID @ 0x1b

NXOS Grub <<<<< Ctrl+C here
disable watchdog
Aborting config file read and autoboot
No autoboot or failed autoboot. falling to loader

Loader Version 1.12

loader >

In the this example, the Secondary Supervisor started booting to 10.3(2)F then recognized itself as the Standby Supervisor and rebooted into loader > prompt.
Cisco N9800(R) Series BIOS Ver 1.11 Primary
Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz
Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU9
Chipguard Init
TAM Library Version: 3.4.26
Aikido Version: 5.0.13
Aikido Chip Serial Number:
006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000
Aikido FPGA Version: 0x120
Aikido chip authentication successful
Aikido KEY Format: UEFI
Cisco PK Key
Cisco Key Key
Cisco DBX Key
Cisco OS Rel Key
Cisco OS Dev Key

Board type 5
IOFPGA @ 0xb2400000
SLOT_ID @ 0x1c

NXOS Grub
disable watchdog
Filesystem type unknown, partition type 0x83
check_bootmode: grub: Continue grub
Trying to read config file /boot/grub/menu.lst.local from (hd0,4)
Filesystem type is ext2fs, partition type 0x83

Booting bootflash:/nxos64-cs.10.3.2.F.bin ...
Booting bootflash:/nxos64-cs.10.3.2.F.bin <<<
Trying diskboot
Filesystem type is ext2fs, partition type 0x83
Image valid

Image Signature verification was Successful.

Boot Time: 7/23/2024 22:49:10
mount: overlay mounted on /newroot/usr.
Installing klm_card_index
done
Setting nativeboot
Linking n9k mando SSD partition...
creating flash devices BOOT_DEV= sda
INIT: version 2.88 booting
Installing ata_piix module ... done.
Installing kheaders module ... done.
Unsquashing rootfs ...
Total size needed in bootflash is 153632
check bootflash : OK
Total size needed in bootflash is 54464
check bootflash : OK
Enabling 8250 serial driver spurious INTs workaround
Installing isan procfs ... done.
is_lxc: is_titan_eor: is_stby: suffix: klm_ftrace: /isanboot/lib/modules/klm_ftrace.o
Installing ftrace in non-lxc mode done
Installing SSE module with card index 21213 ... done.
Creating SSE device node 243 ... done.
Executing /etc/rc.d/rcS.d/S06exablaze start
Executing /etc/rc.d/rcS.d/S06exablaze start
Using cctrli_lite on Mando SUP...
Loading I2C driver ... done.
Installing CCTRL driver for card_type 132 without NEED_GEM ... done.
Loading IGB driver ...
Loading Intel igb driver for Mando...14.81: \nInstalling klm_ins_ixgbe for Mando... card_index=21213

Installing klm_ins_ixgbe ... done.
Not Micron SSD...

Checking all filesystems.
Extracting rpms from image...
/
MANDO RP CPA setup done
Installing SPROM driver ... 21213 IS_N9K done.
@@@ MANDO-RP
Installing pfmsvcs module ...done.
Installing nvram module ... done.
Installing if_index module with port mode 6 ... done.
Installing fcfwd
Installing RNI lcnd ... done.
Installing lcnd ... done.
\n S090setup-eth sup_start
autoneg unmodified, ignoring
no pause parameters changed, aborting
autoneg unmodified, ignoring
no pause parameters changed, aborting
Installing psdev ...
Installing veobc module ... done.
RTNETLINK answers: File exists
Clean up previous pcap files present in tmp directory
Checking SR card
Card Index is 21213
48.15: Inserting obfl & mtd spi driver ...
old data: 0 new data: 9f000000
old data: 0 new data: 0
old data: 20907001 new data: a0901406
[ 48.207892] fpga_setup_probe
48.26: Done..Inserting obfl & mtd spi driver ...
Making OBFL character devices
old data: c000000 new data: 9f000000
old data: 10c0000 new data: 0
old data: 989040 new data: a0901406
mounting plog for N9k!
48.31: mounting plog for N9k!
48.32: Done..mounting plog for N9k!
mounting recovery for N9k!
48.33: mounting recovery for N9k!
Mounting OBFL pstore for mtd
Inserting kernel_services module ... done.
Making kernel_services character devices
cgroups initialized
update-alternatives: Linking /usr/bin/unshare to /usr/bin/unshare.util-linux
Removing any system startup links for cgroups-init ...
Adding system startup for /etc/init.d/cgroups-init.
Running groupadd commands...
NOTE: docker-ce: Performing groupadd with [ -r docker]
update-alternatives: Linking /bin/vi to /usr/bin/vim.tiny
update-alternatives: Linking /usr/bin/vim to /usr/bin/vim.tiny
Removing bootvar to force RP(standby) to be at loader promptYou can use cmdline enable_standby to allow RP(standby) to boot
INIT: Switching/isan/etc/rc.cleanup ignoring inittab invocations
Stoping third party daemons
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Unmounting filesystems...

Cisco N9800(R) Series BIOS Ver 1.11 Primary
Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz
Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU9
Chipguard Init
TAM Library Version: 3.4.26
Aikido Version: 5.0.13
Aikido Chip Serial Number:
006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000
Aikido FPGA Version: 0x120
Aikido chip authentication successful
Aikido KEY Format: UEFI
Cisco PK Key
Cisco Key Key
Cisco DBX Key
Cisco OS Rel Key
Cisco OS Dev Key

Board type 5
IOFPGA @ 0xb2400000
SLOT_ID @ 0x1c

NXOS Grub
disable watchdog
Filesystem type unknown, partition type 0x83
check_bootmode: grub: Continue grub
Trying to read config file /boot/grub/menu.lst.local from (hd0,4)
Filesystem type is ext2fs, partition type 0x83
Trying to read config file /boot/grub/menu.lst.local from (hd0,5)
Filesystem type is ext2fs, partition type 0x83
No autoboot or failed autoboot. falling to loader

Loader Version 1.11

loader >

To initiate image sync with the Active Supervisor to establish HA, we have 2 methods we can use depending on our BIOS version.

Image Sync

BIOS Version

You will need to determine the current BIOS version of the Secondary Supervisor.
The current version can be found on the 1st line of initial output when the supervisor is inserted & after reboot before the loader > prompt.

Cisco N9800(R) Series BIOS Ver 1.11 Primary <<<
Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz
Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU9
Chipguard Init
TAM Library Version: 3.4.26
Aikido Version: 5.0.13
Aikido Chip Serial Number:
006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000
Aikido FPGA Version: 0x120
Aikido chip authentication successful
Aikido KEY Format: UEFI
Cisco PK Key
Cisco Key Key
Cisco DBX Key
Cisco OS Rel Key
Cisco OS Dev Key
Board type 5
IOFPGA @ 0xb2400000
SLOT_ID @ 0x1c
NXOS Grub

Force Download from Active

If the BIOS version installed is 1.11 or HIGHER, image sync can be manually initiated to the Secondary Supervisor from the Active Supervisor using the reload module 28 force-dnld command.

The Secondary Supervisor must be at the loader > prompt when executing this command.

Similar to the Different Release Images section, the Secondary Supervisor will be rebooted to force NetBoot and download the Active Supervisor's boot image to the Secondary Supervisor.

From the Active Supervisor -

N9K-C9808# reload module 28 force-dnld
N9K-C9808# 2024 Jul 23 22:59:15 N9K-C9808 %$ VDC-1 %$ %PLATFORM-2-PFM_MODULE_RESET: Manual restart of Module 28 from Command Line Interface

From the Secondary Supervisor -

Loader Version 1.11

loader >

< Reload Initiated by Active - Reboots into NetBoot process >
Cisco N9800(R) Series BIOS Ver 1.11 Primary
Intel(R) Xeon(R) CPU D-1530 @ 2.40GHz
Board Type 0x220 PID N9K-C9808 Serial FOX2739PFU4
Chipguard Init
TAM Library Version: 3.4.26
Aikido Version: 5.0.13
Aikido Chip Serial Number:
006FE1619905AB2897EB0DE352DFEFF300000000000000000000000000000000
Aikido FPGA Version: 0x120
Aikido chip authentication successful
Aikido KEY Format: UEFI
Cisco PK Key
Cisco Key Key
Cisco DBX Key
Cisco OS Rel Key
Cisco OS Dev Key
Board type 5
IOFPGA @ 0xb2400000
SLOT_ID @ 0x1c
NXOS Grub
disable watchdog
Finding driver for NIC vendor 8086 Device 15ab
i210 devide id = 1537
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
x check driver for NIC vendor 8086 Device 15ab
inside driver for NIC vendor 8086 Device 15ab
pci_base_addres 0= b2c0000c ioaddr= 0
mask issue
Found the device 8086:15ab at ioaddr 1000, membase b2c00000 at 4:0
Probing...
intelx probe
eobc MAC address 00000000:00000000:00000000:0000001c:00000000:00000000
board_type=5
network is ready.
bp_op = 1
bp_htype = 1
bp_hlen = 6
bp_xid = -160883712
bp_hwaddr = 00:00:00:1C:00:00
bp_hops = 0
bp_secs = 2132726648
*retry 1 currticks 27126 timeout = 720
*retry 2 currticks 27684 timeout = 720
bp_op = 1
bp_htype = 1
bp_hlen = 6
bp_xid = -160883712
bp_hwaddr = 00:00:00:1C:00:00
bp_hops = 0
bp_secs = 2132726648
errnum = 0
reqretry = 0
*2 retry 2 currticks 27684 timeout = 720
Filename: /img-sync/curr-ks.img
board_type=5
board_type=5
Address: 127.1.1.28
Netmask: 255.255.0.0
Server: 127.1.1.27
Gateway: 0.0.0.0
Filesystem type is tftp, using whole disk
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Image valid

Image Signature verification was Successful.

Once the download has complete, the Secondary Supervisor will boot to the new image then move to establish ha-standby state with the Active Supervisor.

N9K-C9808# show module
Mod Ports Module-Type Model Status
--- ----- ------------------------------------------------ --------------------- --------
27 0 Supervisor Module N9K-C9800-SUP-A active *
28 0 Supervisor Module N9K-C9800-SUP-A ha-standby

Mod Sw Hw Slot
--- ----------------------- ------ ----
27 10.4(3) 1.0 SUP1
28 10.4(3) 1.0 SUP2

EOBC Boot Secondary Supervisor

If the BIOS version installed is LOWER than 1.11, image sync can ONLY be manually initiated by the Secondary Supervisor from within the loader > prompt using the eobc command.

This will force the switch to TFTP boot from the active supervisor then sync with it.

loader > ?
? Print the command list
boot Boot image
bootmode Display/Change current boot mode
dir List file contents on a device
eobc Booting image from active sup via EOBC channel
keyinfo BIOS KEY information
help Print the command list or the specific command usage
ip Setting IP address or gateway address
reboot Reboot the system
serial Serial console setting
set Set network configuration
show Show loader configuration

loader > eobc
Finding driver for NIC vendor 8086 Device 15ab
i210 devide id = 1537
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
loop driver for NIC vendor 8086 Device 15ab
x check driver for NIC vendor 8086 Device 15ab
inside driver for NIC vendor 8086 Device 15ab
pci_base_addres 0= b2c0000c ioaddr= 0
mask issue
Found the device 8086:15ab at ioaddr 1000, membase b2c00000 at 4:0
Probing...
intelx probe

< After a few minutes supervisor starts TFTP Boot process >

Address: 127.1.1.27
Netmask: 255.255.0.0
Server: 127.1.1.28
Gateway: 0.0.0.0
Filesystem type is tftp, using whole disk
................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Image valid

< After some time the TFTP boot process completes and the Image Sync process begins. >

Saving image for img-sync ...
Enough free disk space found on bootflash

< The sync process does not show progress status and can take some time. >

VERIFYING Md5 checksum for /bootflash/curr-ks.img ....
Verify OK!!

< Soon afterwards the Supervisor successfully completes the boot and sync process then comes up as the Standby Supervior. >

ethernet switching mode Fri Jun 21 15:46:14 UTC 2024
prepare span CLI
[ 666.302971] MANDO RDN : register_rdn_int
[ 666.307460] register_rdn_int : Sent First Heartbeat

User Access Verification
N9K-C9808(standby) login:

The entire boot and sync process can take more than 15 minutes.
It can be monitored from the console.

On the Active Supervisor you can confirm the status of the Secondary Supervisor is ha-standby.

N9K-C9808# show module
Mod Ports Module-Type Model Status
--- ----- ------------------------------------------------ --------------------- --------
27 0 Supervisor Module N9K-C9800-SUP-A active *
28 0 Supervisor Module N9K-C9800-SUP-A ha-standby

Mod Sw Hw Slot
--- ----------------------- ------ ----
27 10.4(3) 1.0 SUP1
28 10.4(3) 1.0 SUP2

Performing either method mentioned is only required one time.

It is recommended to delete the 10.3(x) release from bootflash of the Standby Supervisor leaving the 10.4(x) release remaining.

Upgrade BIOS of Standby Supervisor

The Active Supervisor will already be on the most up to date BIOS version if a supported upgrade doing install all nxos was done on it.

The processes shown in the Different Release Images, Force Download from Active and EOBC Boot Secondary Supervisor sections do NOT upgrade the BIOS on the Standby Supervisor.

The next step is to upgrade the BIOS on the Standby Supervisor to match the Active.

If just the Standby Supervisor BIOS is the only one being upgraded it does not need to be reloaded.

N9K-C9808# install all nxos bootflash:nxos64-cs.10.4.3.F.bin
Installer will perform compatibility check first. Please wait.
Installer is forced disruptive

Verifying image bootflash:/nxos64-cs.10.4.3.F.bin for boot variable "nxos".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos64-cs.10.4.3.F.bin.
[####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos64-cs.10.4.3.F.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS

Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
27 yes disruptive copy-only default upgrade is not hitless
28 yes disruptive none default upgrade is not hitless

Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
27 nxos 10.4(3) 10.4(3) no
27 bios v01.09(12/19/2022):v01.03(12/29/2021) v01.12(11/28/2023) yes
28 nxos 10.4(3) 10.4(3) no
28 bios v01.12(11/28/2023):v01.03(12/29/2021) v01.12(11/28/2023) no

Switch will be reloaded for disruptive upgrade.
Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.

Performing runtime checks.
[####################] 100% -- SUCCESS

Syncing image bootflash:/nxos64-cs.10.4.3.F.bin to standby.
[####################] 100% -- SUCCESS

Setting boot variables.
[####################] 100% -- SUCCESS

Performing configuration copy.
[####################] 100% -- SUCCESS

Module 20: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 22: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 24: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 26: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 27: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Module 28: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Install has been successful.

BIOS version on both Standby and Active Supervisors now match but EPLD version does not:
N9K-C9808(standby)# show hardware internal dev-version
-------------------------------------------------------------------
Name InstanceNum Version Date
-------------------------------------------------------------------
IO FPGA 0 0x1001b
TM FPGA 0 0x10006
BIOS version v01.12(11/28/2023)
Alternate BIOS version v01.03(12/29/2021)

N9K-C9808# show hardware internal dev-version
-------------------------------------------------------------------
Name InstanceNum Version Date
-------------------------------------------------------------------
IO FPGA 0 0x10017
TM FPGA 0 0x10002
FM Module-20 FPGA 1 0x10000
FM Module-22 FPGA 3 0x10000
FM Module-24 FPGA 5 0x10000
FM Module-26 FPGA 7 0x10000
BIOS version v01.12(11/28/2023)
Alternate BIOS version v01.03(12/29/2021)

Upgrade EPLD of Supervisors

As mentioned in the Cisco Nexus 9000 Series NX-OS High Availability and Redundancy Guide, Release 10.4(x), it is a mandatory requirement to perform an EPLD upgrade for Nexus 9800 Dual Supervisor deployments.

As further elaborated in Cisco bug ID CSCwk42621 This table shows the minimum EPLD version on the Supervisors for HA to fully function:

You can upgrade individual modules or the entire chassis at once.
Depending on which modules are upgraded, this procedure can be disruptive.

It is recommended to upgrade all modules if possible.

The number of modules being upgraded will affect how long the entire upgrade will take.

Cisco recommends performing the EPLD upgrade during a maintenance window to minimize potential disruption.

N9K-C9808# install epld bootflash:n9000-epld.10.4.3.F.img module ?
WORD X or x-y (Max Size 5)
all Install all the modules

N9K-C9808# install epld bootflash:n9000-epld.10.4.3.F.img module all
Compatibility check:
Module Type Upgradable Impact Reason
------ ----------------- ---------- ---------- ------
20 FM Yes disruptive Module Upgradable
22 FM Yes disruptive Module Upgradable
24 FM Yes disruptive Module Upgradable
26 FM Yes disruptive Module Upgradable
27 SUP Yes disruptive Module Upgradable
28 SUP Yes disruptive Module Upgradable

Retrieving EPLD versions.... Please wait.
Images will be upgraded according to following table:
Module Type EPLD Running-Version New-Version Upg-Required
------ ---- ------------- --------------- ----------- ------------
20 FM MI FPGA 0x10000 0x10002 Yes
22 FM MI FPGA 0x10000 0x10002 Yes
24 FM MI FPGA 0x10000 0x10002 Yes
26 FM MI FPGA 0x10000 0x10002 Yes
27 SUP TM FPGA 0x10006 0x10006 No
27 SUP IO FPGA 0x1001b 0x10020 Yes
28 SUP TM FPGA 0x10002 0x10006 Yes
28 SUP IO FPGA 0x10017 0x10020 Yes
The above modules require upgrade.
Do you want to continue (y/n) ? [n] y

Do you want to upgrade the Active Supervisor? [n] y

Proceeding to upgrade Modules.

Proceeding to upgrade Modules.

Starting Module 20 EPLD Upgrade
Module 20 : MI FPGA [Programming] : 0.00% ( 0 of 64 sectors)
Module 20 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)
Module 20 EPLD upgrade is successful..

Module 20 EPLD upgrade is successful.

Starting Module 22 EPLD Upgrade
Module 22 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)
Module 22 EPLD upgrade is successful..

Module 22 EPLD upgrade is successful.

Starting Module 24 EPLD Upgrade
Module 24 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)
Module 24 EPLD upgrade is successful..

Module 24 EPLD upgrade is successful.

Starting Module 26 EPLD Upgrade
Module 26 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)
Module 26 EPLD upgrade is successful..

Module 26 EPLD upgrade is successful.

Module 27 : IO FPGA [Programming ] : 0.00% ( 0 of 64 total sectors)
Module 27 : IO FPGA [Programming ] : 100.00% ( 64 of 64 total sectors)
Module 27 EPLD upgrade is successful.

Starting Module 28 EPLD Upgrade
Module 28 : IO FPGA [Programming] : 100.00% ( 64 of 64 sectors)
Module 28 EPLD upgrade is successful..
Module Type Upgrade-Result
------ ------------------ --------------
27 SUP Success
28 SUP Success

EPLDs upgraded. Performing switchover.

Module 28 EPLD upgrade is successful.

Reloading the Chassis...
Reseting Standby SUP (Module 27) FPGAs. Please wait...
Module 27 will reload.
Resetting Module 20 FPGAs. Please wait...
Module 20 will reload.
Resetting Module 22 FPGAs. Please wait...
Module 22 will reload.
Resetting Module 24 FPGAs. Please wait...
Module 24 will reload.
Resetting Module 26 FPGAs. Please wait...
Module 26 will reload.
Processing Active sup reset...
Reseting Active SUP (Module 28) FPGAs. Please wait...
Reload in 10 seconds .....
Resetting Module 20 FPGAs. Please wait...
Module 20 will reload.

After EPLD upgrade and reload the Active and Standby Supervisor roles can be switched:
N9K-C9808# show module
Mod Ports Module-Type Model Status
--- ----- ------------------------------------------------ --------------------- --------
27 0 Supervisor Module N9K-C9800-SUP-A active *
28 0 Supervisor Module N9K-C9800-SUP-A ha-standby

To change the active supervisor you can use the "system switchover" command to reload the Active Supervisor and make the Standby Supervisor the Active.
From the Active Supervisor:
N9K-C9808# system switchover

From the Standby Supervisor as it becomes the Active:
N9K-C9808(standby) login: [ 784.909143] Got RP P2PM intr, Becoming Active !!

User Access Verification
N9K-C9808 login: admin
Password:

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2024, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source. This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0 or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
N9K-C9808# show module
Mod Ports Module-Type Model Status
--- ----- ------------------------------------------------ --------------------- --------
27 0 Supervisor Module powered-up
28 0 Supervisor Module N9K-C9800-SUP-A active *

Summary

1. Install Secondary Supervisor

2. Perform process to image sync Secondary with Active Supervisor depending on pre-installed release

3. Upgrade BIOS on Standby Supervisor

4. Upgrade EPLD for all modules