stumbling blocks in upgrading Cisco UCS (Part 1of2)

stumbling blocks in upgrading Cisco UCS (Part 1of2)

In theory a upgrade of an UCS is not complicated, but there are many stumbling blocks.

I show you my procedure, bugs, some hints and Cisco TAC Support stuff 😉


  • See how i lost the Fibrechannel link permanently as a result of this “little” upgrade
  • See some hints to upgrade successfully

First of all, read carefully the release notes and check the compatibility lists!

As an example, we have an 4 year old Blade B230M2 with an Adapter M81KR (N20-AC0002) that is deprecated and not supported in the latest 3.1 Release! 3.1 Relase Notes

For that reason i show you the upgrade from version 2.2(3d) to 2.2(7c)


My procedure is the following:

  1. Read Release Notes carefully
  2. Check compatibility issues for Hardware or Firmware Versions
  3. Check space on the Fabric Interconnect
  4. Check filling level of the temp directory on the Fabric Interconnect (not more that 10%)
  5. Check reboot issue in Firmware 2.2(3d) BUG63966
  6. Check scheduled Backup or create it
  7. Upload the Bundles
  8. Check Cluster State
  9. Disable Home Call
  10. Check Datapaths & Note for later comparsion
  11. Activate UCS Manager
  12. Update IO Modules (Subordinate)
  13. Activate IO Modules (Subordinate)
  14. Activate FI (Subordinate)
  15. Check Datapaths if all are up again
  16. Halftime, take a Coffe or one Beer 😉
  17. Failover FI on CLI
  18. Check Datapaths
  19. Update IO Modules (Now Subordinate)
  20. Activate IO Modules (Now Subordinate)
  21. Activate FI (Now Subordinate)
  22. Check Datapaths
  23. Upgrade Servers/Blades
  24. Take some more Beer 😉

Let’s start …

Check space on the Fabric Interconnect


Check filling level of the temp directory on the Fabric Interconnect

You need this only if the last reboot of your FI is long ago (>1 Year,)

In my case /var/tmp is 10% filled up -> too much for upgrade!

Attention, i recommend that TAC Support do that for you! You also need that debug-plugin from Cisco to do this.

UCS2-B# connect local-mgmt b
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

UCS2-B(local-mgmt)# cd debug_plugin/
UCS2-B(local-mgmt)# copy ucs-dplug.5.2.3.N2.2.22c.gbin x
UCS2-B(local-mgmt)# load x
###############################################################
  Warning: debug-plugin is for engineering internal use only!
  For security reason, plugin image has been deleted.
###############################################################
Successfully loaded debug-plugin!!!
Linux(debug)# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root             300M  180M  121M  60% /
none                  2.0M  4.0K  2.0M   1% /post
none                  300M  180M  121M  60% /var
none                  3.0G 1008M  2.0G  34% /isan
none                  600M   57M  544M  10% /var/tmp
none                  2.0G   69M  1.9G   4% /var/sysmgr
none                  140M   72K  140M   1% /var/sysmgr/ftp
none                  128M     0  128M   0% /callhome
none                  512M  281M  232M  55% /dev/shm
none                  240M   77M  164M  32% /volatile
none                  8.0M     0  8.0M   0% /debug
/dev/mtdblock2         48M  1.4M   47M   3% /mnt/plog
/dev/sda5              84M  4.3M   75M   6% /mnt/cfg/0
/dev/sda6              84M  4.3M   75M   6% /mnt/cfg/1
none                  400M  4.5M  396M   2% /var/sysmgr/startup-cfg
/dev/sda4              84M  7.3M   72M  10% /mnt/pss
/dev/sda3              16G  6.8G  8.4G  45% /bootflash
/dev/sda7             3.8G   75M  3.6G   3% /opt
/dev/mtdblock3        3.9M  2.2M  1.6M  58% /opt/db/nvram
/dev/sda8             3.8G   35M  3.6G   1% /workspace
/dev/sda9             5.7G   33M  5.3G   1% /spare
tmpfs                 1.0M   40K  984K   4% /bootflash/sysdebug/tftpd_logs
none                   50M  6.4M   44M  13% /debug-plugin
Linux(debug)# cd /var/tmp
Linux(debug)# ls -lSr
total 57328
-rw-r--r-- 1 root   root          0 Jan  9  2015 vsh.config.log
-rwxrwxrwx 1 root   root          0 Jan  9  2015 sshk_out
-rwxrwxrwx 1 root   root          0 Jan  9  2015 sshk_err
srwxrwxrwx 1 root   root          0 Jun 11 07:55 sfcbLocalSocket
-rwxrwxrwx 1 root   root          0 Jun 13 10:47 security_stdout
-rwxrwxrwx 1 root   root          0 Jun 13 10:47 security_stderr
-rw-rw-rw- 1 root   root          0 Jan  9  2015 npv_fa_pending.trm
-rw-rw-rw- 1 root   root          0 Jan  9  2015 npv_fa_pending.map
-rw-rw-rw- 1 root   root          0 Jan  9  2015 npv_fa_pending.dat
---------- 1 root   root          0 Jan  9  2015 npv_cfs_multicast
---------x 1 root   root          0 Jun 13 13:36 ni-port-stats.log
-rw-rw-rw- 1 root   root          0 Jan  9  2015 mvsh.4997
-rw-rw-rw- 1 root   root          0 Jan  9  2015 mgmtif.lock
-rw-rw-rw- 1 root   root          0 Jun 13 13:40 lockcisco
-rw-rw-rw- 1 root   root          0 Jan  9  2015 igmp_restart.log
-rw-rw-rw- 1 root   root          0 Mar  5 18:54 httpd_info_diff
-rw-rw-rw- 1 root   root          0 Jun 11 07:55 httpd_cimc.sh.log
-rw-rw-rw- 1 root   root          0 Jan  9  2015 fwm1.out
-rw-rw-rw- 1 root   root          0 Jan  9  2015 fcoe_mgr
-rw-rw-rw- 1 root   root          0 Jan  9  2015 cmp_slot_id.1
-rw-rw-rw- 1 root   root          0 Jun 13 10:43 bootloader_ver
-rw-r--r-- 1 root   root          0 Sep  1  2015 RewriteCimc.log
-rw-r--r-- 1 root   root          0 Jan  9  2015 Rewrite.log
-rw-rw-rw- 1 root   root          0 Jun 13 13:41 LOG_EXPORT_CONFIG.LCK
-rw-rw-rw- 1 root   root          0 Jun 11 07:54 CIMCWEBSVC.LCK
-rw-rw-rw- 1 root   root          8 Jan  9  2015 fc_nport_mux
lrwxrwxrwx 1 root   root         13 Jun 13 13:41 ucsshlog.lock -> /tmp/ucsshlog
-rw-r--r-- 1 root   root         16 Jan  9  2015 get_swname
-rw-rw-rw- 1 root   root         18 Jun 11 07:55 samcli.boot.setup
-rw-r--r-- 1 root   root         18 Jan  9  2015 initial.config.setup
-rw-r--r-- 1 root   root         18 Jan  9  2015 initial.config.name.setup
-rw-rw-rw- 1 root   root         20 Jun 11 07:54 samcli.unload
lrwxrwxrwx 1 root   root         20 Jan  9  2015 logs -> /var/sysmgr/tmp_logs
-rw-rw-rw- 1 root   root         24 Jan  9  2015 SC_port-ag_child_intfsubmode_success
-rw-rw-rw- 1 root   root         24 Jan  9  2015 SC_dcos-ag_child_intfsubmode_success
-rw-rw-rw- 1 root   root         26 Jan  9  2015 SC_extvmm-ag_child_intfsubmode_success
-rw-r--r-- 1 root   root         27 Jun 11 07:55 sam_upgrade_samcli.log
-rw-rw-rw- 1 root   root         28 Jan  9  2015 SC_sw-stats-ag_child_intfsubmode_success
-rw-rw-rw- 1 root   root         31 Jan  9  2015 evmc.5004
lrwxrwxrwx 1 root   root         34 Jan  9  2015 sam_init_virt_plugin_complete -> /tmp/sam_init_virt_plugin.out.5362
-rw-rw-rw- 1 root   root         38 Jun 11 10:37 snmp.log
-rw-r--r-- 1 root   root         38 Jun 11 07:55 ldap-debug-sam.txt
drwxrwxrwx 2 root   root         40 Jan  9  2015 vsh
drwxrwxrwx 2 root   root         40 Jan  9  2015 callhome
-rw-r--r-- 1 root   root         55 Jan  9  2015 mgmtif.tmp.1899
-rw-r--r-- 1 root   root         56 Jan  9  2015 boot_uptime.log
-rw-rw-rw- 1 root   root         68 Jun 13 10:44 m2rib_ascii.5287
-rw-rw-rw- 1 root   root         78 Jun 11 07:55 ldap-debug.txt
drwxrwxrwx 2 root   root         80 Jun 10 15:19 anonymous
-rwxrwxrwx 1 root   root        103 Jun 11 07:56 security_tmp_script
-rw-rw-rw- 1 root   root        114 Jan  9  2015 fsrv_vsh_sdb
-rw-r--r-- 1 root   root        120 Jan  9  2015 vsh.switchname.config.log
-rw-r--r-- 1 root   root        122 Jan  9  2015 startup.patchsnmpdconf.log
-rw-r--r-- 1 root   root        137 Jun 11 10:37 openportinfo.txt
-rw-rw-rw- 1 root   root        161 Jan  9  2015 fc-pcm-log-5360
-rw-rw-rw- 1 root   root        166 Jan  9  2015 cfs_mac.log
-rw-rw-rw- 1 root   root        187 Jan  9  2015 sam_init_virt_plugin.out.6351
-rw-rw-rw- 1 daemon shadow      187 Mar 21 15:03 sam_init_virt_plugin.out.11793
-rw-rw-rw- 1 daemon shadow      187 Nov  1  2015 sam_init_virt_plugin.out.10358
-rw-rw-rw- 1 daemon shadow      189 Jun 11 07:57 sam_init_virt_plugin.out.3602
-rw-rw-rw- 1 root   root        189 Jun 11 07:55 sam_init_virt_plugin.out.31414
-rw-rw-rw- 1 root   root        189 Jun 11 07:55 sam_init_virt_plugin.out.31408
-rw-rw-rw- 1 root   root        189 Jun 11 07:55 sam_init_virt_plugin.out.31406
-rw-rw-rw- 1 root   root        189 Jun 11 07:55 sam_init_virt_plugin.out.31399
-rw-rw-rw- 1 root   root        189 Jun 11 07:55 sam_init_virt_plugin.out.31391
-rw-rw-rw- 1 root   root        189 Jun 11 07:55 sam_init_virt_plugin.out.31337
-rw-rw-rw- 1 root   root        189 Jun 11 07:55 sam_init_virt_plugin.out.31323
-rw-rw-rw- 1 root   root        189 Jun 11 07:55 sam_init_virt_plugin.out.31321
-rw-rw-rw- 1 root   root        232 Jan  9  2015 radius_debug.log
-rw-rw-rw- 1 root   root        260 Jan  9  2015 npv_fa_pending.pss
-rw-rw-rw- 1 root   root        270 Jan  9  2015 nsmgr_temp.txt
-rw-rw-rw- 1 root   root        293 Jun 11 07:55 samcli.boot.log
-rw-rw-rw- 1 root   root        345 Jan  9  2015 sam_init_virt_plugin.out.5362
-rw-r--r-- 1 root   root        405 Jan  9  2015 sam_startup.log
-rw-rw-rw- 1 root   root        449 Jan  9  2015 ldap_debug.log
-rw-rw-rw- 1 root   root        486 Jan  9  2015 aaa_debug.log
-rw-rw-rw- 1 root   root        580 Jan  9  2015 security_debug.log
-rw-rw-rw- 1 root   root        694 Jan  9  2015 lcc.log
-rw-r--r-- 1 root   root        792 Jan  9  2015 mysrg
-rw-rw-rw- 1 root   root        810 Jan  9  2015 tacacs_crypt_service_1
-rw-r--r-- 1 root   root       1050 Jun 11 07:55 startup.catalogfile.log
-rw-rw-rw- 1 root   root       1386 Jun 13 13:37 ni-port-stats.txt
-rw-rw-rw- 1 root   root       1592 Jun 11 07:55 pstate
-rw-rw-rw- 1 root   root       1851 Mar  5 11:13 cisco_nexus_1000v_extension_0.xml
-rw-rw-rw- 1 root   root       2022 Jun 11 10:37 radius_crypt_service_1
-rw-rw-rw- 1 root   root       2177 Jan  9  2015 sam_init_virt_plugin.out.5443
-rw-rw-rw- 1 root   root       2177 Jan  9  2015 sam_init_virt_plugin.out.5424
-rw-rw-rw- 1 root   root       2177 Jan  9  2015 sam_init_virt_plugin.out.5423
-rw-rw-rw- 1 root   root       2177 Jan  9  2015 sam_init_virt_plugin.out.5420
-rw-rw-rw- 1 root   root       2362 Jan  9  2015 sam_init_virt_plugin.out.5356
-rw-rw-rw- 1 root   root       2547 Jan  9  2015 sam_init_virt_plugin.out.5380
-rw-rw-rw- 1 root   root       2706 Jan  9  2015 _clis_parse_tree_.log
-rw-r--r-- 1 root   root       2741 Jan  9  2015 bond_setup.log
-rw-rw-rw- 1 root   root       2883 Jun 13 10:44 sksd_crypt_service_1
-rw-rw-rw- 1 root   root       4099 Jan  9  2015 stp.log.1
-rwxr-xr-x 1 root   root       4285 Jun 11 10:37 iptab.sh
-rw-rw-rw- 1 root   root       5241 Jan  9  2015 first_setup.log
-rw-rw-rw- 1 root   root       8456 Dec 17 15:17 dave.log
-rw-rw-rw- 1 root   root      10748 Jun 11 07:55 snmpd-debug-sam.txt
-rw-rw-rw- 1 root   root      17607 Jan  9  2015 fcoe_mgr_init.log
-rw-rw-rw- 1 root   root      21356 Jan  9  2015 fm_debug.log.vdc_1.4927
-rw-rw-rw- 1 root   root      62198 Jun 11 07:55 syslogd_errlog.4894_4894
-rw-rw-rw- 1 root   root     174261 Jun 13 13:37 afm_srv_15.log
-rw-r--r-- 1 root   root     200394 Jan  9  2015 sam_bridge_boot.log
-rw-rw-rw- 1 root   root     250034 Jun 11 03:42 afm_srv_15.log.0
-rw-rw-rw- 1 root   root   57569165 Jun 13 13:36 smm.log
Linux(debug)# cp /dev/null /var/tmp/smm.log
Linux(debug)# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root             300M  180M  121M  60% /
none                  2.0M  4.0K  2.0M   1% /post
none                  300M  180M  121M  60% /var
none                  3.0G 1008M  2.0G  34% /isan
none                  600M  1.1M  599M   1% /var/tmp
none                  2.0G   69M  1.9G   4% /var/sysmgr
none                  140M   72K  140M   1% /var/sysmgr/ftp
none                  128M     0  128M   0% /callhome
none                  512M  281M  232M  55% /dev/shm
none                  240M   77M  164M  32% /volatile
none                  8.0M     0  8.0M   0% /debug
/dev/mtdblock2         48M  1.4M   47M   3% /mnt/plog
/dev/sda5              84M  4.3M   75M   6% /mnt/cfg/0
/dev/sda6              84M  4.3M   75M   6% /mnt/cfg/1
none                  400M  4.5M  396M   2% /var/sysmgr/startup-cfg
/dev/sda4              84M  7.3M   72M  10% /mnt/pss
/dev/sda3              16G  6.8G  8.4G  45% /bootflash
/dev/sda7             3.8G   75M  3.6G   3% /opt
/dev/mtdblock3        3.9M  2.2M  1.6M  58% /opt/db/nvram
/dev/sda8             3.8G   35M  3.6G   1% /workspace
/dev/sda9             5.7G   33M  5.3G   1% /spare
tmpfs                 1.0M   40K  984K   4% /bootflash/sysdebug/tftpd_logs
none                   50M  6.4M   44M  13% /debug-plugin
Linux(debug)# exit
exit

Check reboot issue in Firmware 2.2(3d) BUG63966

Another Bug we ran into some months ago 🙁

The problem is, a config save during shutdown or reboot maybe can’t happen because the system stops before the save is complete. -> Cisco’s resolution is to append a sleeptimer 😉

Here is how Cisco TAC Support fixed that:

UCS2-B# connect local-mgmt b
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

UCS2-B(local-mgmt)# cd debug_plugin/
UCS2-B(local-mgmt)# copy ucs-dplug.5.2.3.N2.2.22c.gbin x
UCS2-B(local-mgmt)# load x
###############################################################
  Warning: debug-plugin is for engineering internal use only!
  For security reason, plugin image has been deleted.
###############################################################
Successfully loaded debug-plugin!!!
Linux(debug)# cd /etc/init.d
Linux(debug)# ls
apachecracklibd dhcp functions getresvport.sh halt init-functions iptables killall netconsole netfsnetwork nfs-common nfs-user-server portmap portmap-wrapperracoonrc rdisc services-wrapper single syslog udev-post
Linux(debug)# cd /isan/bn in
Linux(debug)# cat sam_logs.sh 
#!/bin/sh
/bin/mkdir -p /opt
cd /var/sysmgr
/bin/tar -czf /opt/sam_logs.tgz sam_logs

##switch logs
/bin/mkdir -p /bootflash/sw_trace_logs/
cp /var/sysmgr/tmp_logs/bcm_usd.log* /bootflash/sw_trace_logs/
cd /var/sysmgr/
/bin/tar -czf /bootflash/sw_trace_logs/sw_trace_logs_current.tgz sw_trace_logs/{*.current,*.1.gz} 2>/dev/null
/bin/tar -czf /bootflash/sw_trace_logs/sw_trace_logs_before_reboot.tgz sw_trace_logs 2>/dev/null
sync
sleep 10

Sure, you need to do this on both fabrics!


Check Backup


Upload the Bundles

Upload all your Bundles you downloaded from Cisco


Check Cluster Status

UCS-A# show cluster state
Cluster Id: XXXXXXXXXXXXXXX

A: UP, PRIMARY
B: UP, SUBORDINATE

or GUI


Disable Home Call


Check out Part 2

One thought on “stumbling blocks in upgrading Cisco UCS (Part 1of2)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.