stumbling blocks in upgrading Cisco UCS (Part 1of2)
In theory a upgrade of an UCS is not complicated, but there are many stumbling blocks.
I show you my procedure, bugs, some hints and Cisco TAC Support stuff 😉
- See how i lost the Fibrechannel link permanently as a result of this “little” upgrade
- See some hints to upgrade successfully
First of all, read carefully the release notes and check the compatibility lists!
As an example, we have an 4 year old Blade B230M2 with an Adapter M81KR (N20-AC0002) that is deprecated and not supported in the latest 3.1 Release! 3.1 Relase Notes
For that reason i show you the upgrade from version 2.2(3d) to 2.2(7c)
My procedure is the following:
- Read Release Notes carefully
- Check compatibility issues for Hardware or Firmware Versions
- Check space on the Fabric Interconnect
- Check filling level of the temp directory on the Fabric Interconnect (not more that 10%)
- Check reboot issue in Firmware 2.2(3d) BUG63966
- Check scheduled Backup or create it
- Upload the Bundles
- Check Cluster State
- Disable Home Call
- Check Datapaths & Note for later comparsion
- Activate UCS Manager
- Update IO Modules (Subordinate)
- Activate IO Modules (Subordinate)
- Activate FI (Subordinate)
- Check Datapaths if all are up again
- Halftime, take a Coffe or one Beer 😉
- Failover FI on CLI
- Check Datapaths
- Update IO Modules (Now Subordinate)
- Activate IO Modules (Now Subordinate)
- Activate FI (Now Subordinate)
- Check Datapaths
- Upgrade Servers/Blades
- Take some more Beer 😉
Let’s start …
Check space on the Fabric Interconnect
Check filling level of the temp directory on the Fabric Interconnect
You need this only if the last reboot of your FI is long ago (>1 Year,)
In my case /var/tmp is 10% filled up -> too much for upgrade!
Attention, i recommend that TAC Support do that for you! You also need that debug-plugin from Cisco to do this.
UCS2-B# connect local-mgmt b Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2009, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php UCS2-B(local-mgmt)# cd debug_plugin/ UCS2-B(local-mgmt)# copy ucs-dplug.5.2.3.N2.2.22c.gbin x UCS2-B(local-mgmt)# load x ############################################################### Warning: debug-plugin is for engineering internal use only! For security reason, plugin image has been deleted. ############################################################### Successfully loaded debug-plugin!!! Linux(debug)# df -h Filesystem Size Used Avail Use% Mounted on /dev/root 300M 180M 121M 60% / none 2.0M 4.0K 2.0M 1% /post none 300M 180M 121M 60% /var none 3.0G 1008M 2.0G 34% /isan none 600M 57M 544M 10% /var/tmp none 2.0G 69M 1.9G 4% /var/sysmgr none 140M 72K 140M 1% /var/sysmgr/ftp none 128M 0 128M 0% /callhome none 512M 281M 232M 55% /dev/shm none 240M 77M 164M 32% /volatile none 8.0M 0 8.0M 0% /debug /dev/mtdblock2 48M 1.4M 47M 3% /mnt/plog /dev/sda5 84M 4.3M 75M 6% /mnt/cfg/0 /dev/sda6 84M 4.3M 75M 6% /mnt/cfg/1 none 400M 4.5M 396M 2% /var/sysmgr/startup-cfg /dev/sda4 84M 7.3M 72M 10% /mnt/pss /dev/sda3 16G 6.8G 8.4G 45% /bootflash /dev/sda7 3.8G 75M 3.6G 3% /opt /dev/mtdblock3 3.9M 2.2M 1.6M 58% /opt/db/nvram /dev/sda8 3.8G 35M 3.6G 1% /workspace /dev/sda9 5.7G 33M 5.3G 1% /spare tmpfs 1.0M 40K 984K 4% /bootflash/sysdebug/tftpd_logs none 50M 6.4M 44M 13% /debug-plugin Linux(debug)# cd /var/tmp Linux(debug)# ls -lSr total 57328 -rw-r--r-- 1 root root 0 Jan 9 2015 vsh.config.log -rwxrwxrwx 1 root root 0 Jan 9 2015 sshk_out -rwxrwxrwx 1 root root 0 Jan 9 2015 sshk_err srwxrwxrwx 1 root root 0 Jun 11 07:55 sfcbLocalSocket -rwxrwxrwx 1 root root 0 Jun 13 10:47 security_stdout -rwxrwxrwx 1 root root 0 Jun 13 10:47 security_stderr -rw-rw-rw- 1 root root 0 Jan 9 2015 npv_fa_pending.trm -rw-rw-rw- 1 root root 0 Jan 9 2015 npv_fa_pending.map -rw-rw-rw- 1 root root 0 Jan 9 2015 npv_fa_pending.dat ---------- 1 root root 0 Jan 9 2015 npv_cfs_multicast ---------x 1 root root 0 Jun 13 13:36 ni-port-stats.log -rw-rw-rw- 1 root root 0 Jan 9 2015 mvsh.4997 -rw-rw-rw- 1 root root 0 Jan 9 2015 mgmtif.lock -rw-rw-rw- 1 root root 0 Jun 13 13:40 lockcisco -rw-rw-rw- 1 root root 0 Jan 9 2015 igmp_restart.log -rw-rw-rw- 1 root root 0 Mar 5 18:54 httpd_info_diff -rw-rw-rw- 1 root root 0 Jun 11 07:55 httpd_cimc.sh.log -rw-rw-rw- 1 root root 0 Jan 9 2015 fwm1.out -rw-rw-rw- 1 root root 0 Jan 9 2015 fcoe_mgr -rw-rw-rw- 1 root root 0 Jan 9 2015 cmp_slot_id.1 -rw-rw-rw- 1 root root 0 Jun 13 10:43 bootloader_ver -rw-r--r-- 1 root root 0 Sep 1 2015 RewriteCimc.log -rw-r--r-- 1 root root 0 Jan 9 2015 Rewrite.log -rw-rw-rw- 1 root root 0 Jun 13 13:41 LOG_EXPORT_CONFIG.LCK -rw-rw-rw- 1 root root 0 Jun 11 07:54 CIMCWEBSVC.LCK -rw-rw-rw- 1 root root 8 Jan 9 2015 fc_nport_mux lrwxrwxrwx 1 root root 13 Jun 13 13:41 ucsshlog.lock -> /tmp/ucsshlog -rw-r--r-- 1 root root 16 Jan 9 2015 get_swname -rw-rw-rw- 1 root root 18 Jun 11 07:55 samcli.boot.setup -rw-r--r-- 1 root root 18 Jan 9 2015 initial.config.setup -rw-r--r-- 1 root root 18 Jan 9 2015 initial.config.name.setup -rw-rw-rw- 1 root root 20 Jun 11 07:54 samcli.unload lrwxrwxrwx 1 root root 20 Jan 9 2015 logs -> /var/sysmgr/tmp_logs -rw-rw-rw- 1 root root 24 Jan 9 2015 SC_port-ag_child_intfsubmode_success -rw-rw-rw- 1 root root 24 Jan 9 2015 SC_dcos-ag_child_intfsubmode_success -rw-rw-rw- 1 root root 26 Jan 9 2015 SC_extvmm-ag_child_intfsubmode_success -rw-r--r-- 1 root root 27 Jun 11 07:55 sam_upgrade_samcli.log -rw-rw-rw- 1 root root 28 Jan 9 2015 SC_sw-stats-ag_child_intfsubmode_success -rw-rw-rw- 1 root root 31 Jan 9 2015 evmc.5004 lrwxrwxrwx 1 root root 34 Jan 9 2015 sam_init_virt_plugin_complete -> /tmp/sam_init_virt_plugin.out.5362 -rw-rw-rw- 1 root root 38 Jun 11 10:37 snmp.log -rw-r--r-- 1 root root 38 Jun 11 07:55 ldap-debug-sam.txt drwxrwxrwx 2 root root 40 Jan 9 2015 vsh drwxrwxrwx 2 root root 40 Jan 9 2015 callhome -rw-r--r-- 1 root root 55 Jan 9 2015 mgmtif.tmp.1899 -rw-r--r-- 1 root root 56 Jan 9 2015 boot_uptime.log -rw-rw-rw- 1 root root 68 Jun 13 10:44 m2rib_ascii.5287 -rw-rw-rw- 1 root root 78 Jun 11 07:55 ldap-debug.txt drwxrwxrwx 2 root root 80 Jun 10 15:19 anonymous -rwxrwxrwx 1 root root 103 Jun 11 07:56 security_tmp_script -rw-rw-rw- 1 root root 114 Jan 9 2015 fsrv_vsh_sdb -rw-r--r-- 1 root root 120 Jan 9 2015 vsh.switchname.config.log -rw-r--r-- 1 root root 122 Jan 9 2015 startup.patchsnmpdconf.log -rw-r--r-- 1 root root 137 Jun 11 10:37 openportinfo.txt -rw-rw-rw- 1 root root 161 Jan 9 2015 fc-pcm-log-5360 -rw-rw-rw- 1 root root 166 Jan 9 2015 cfs_mac.log -rw-rw-rw- 1 root root 187 Jan 9 2015 sam_init_virt_plugin.out.6351 -rw-rw-rw- 1 daemon shadow 187 Mar 21 15:03 sam_init_virt_plugin.out.11793 -rw-rw-rw- 1 daemon shadow 187 Nov 1 2015 sam_init_virt_plugin.out.10358 -rw-rw-rw- 1 daemon shadow 189 Jun 11 07:57 sam_init_virt_plugin.out.3602 -rw-rw-rw- 1 root root 189 Jun 11 07:55 sam_init_virt_plugin.out.31414 -rw-rw-rw- 1 root root 189 Jun 11 07:55 sam_init_virt_plugin.out.31408 -rw-rw-rw- 1 root root 189 Jun 11 07:55 sam_init_virt_plugin.out.31406 -rw-rw-rw- 1 root root 189 Jun 11 07:55 sam_init_virt_plugin.out.31399 -rw-rw-rw- 1 root root 189 Jun 11 07:55 sam_init_virt_plugin.out.31391 -rw-rw-rw- 1 root root 189 Jun 11 07:55 sam_init_virt_plugin.out.31337 -rw-rw-rw- 1 root root 189 Jun 11 07:55 sam_init_virt_plugin.out.31323 -rw-rw-rw- 1 root root 189 Jun 11 07:55 sam_init_virt_plugin.out.31321 -rw-rw-rw- 1 root root 232 Jan 9 2015 radius_debug.log -rw-rw-rw- 1 root root 260 Jan 9 2015 npv_fa_pending.pss -rw-rw-rw- 1 root root 270 Jan 9 2015 nsmgr_temp.txt -rw-rw-rw- 1 root root 293 Jun 11 07:55 samcli.boot.log -rw-rw-rw- 1 root root 345 Jan 9 2015 sam_init_virt_plugin.out.5362 -rw-r--r-- 1 root root 405 Jan 9 2015 sam_startup.log -rw-rw-rw- 1 root root 449 Jan 9 2015 ldap_debug.log -rw-rw-rw- 1 root root 486 Jan 9 2015 aaa_debug.log -rw-rw-rw- 1 root root 580 Jan 9 2015 security_debug.log -rw-rw-rw- 1 root root 694 Jan 9 2015 lcc.log -rw-r--r-- 1 root root 792 Jan 9 2015 mysrg -rw-rw-rw- 1 root root 810 Jan 9 2015 tacacs_crypt_service_1 -rw-r--r-- 1 root root 1050 Jun 11 07:55 startup.catalogfile.log -rw-rw-rw- 1 root root 1386 Jun 13 13:37 ni-port-stats.txt -rw-rw-rw- 1 root root 1592 Jun 11 07:55 pstate -rw-rw-rw- 1 root root 1851 Mar 5 11:13 cisco_nexus_1000v_extension_0.xml -rw-rw-rw- 1 root root 2022 Jun 11 10:37 radius_crypt_service_1 -rw-rw-rw- 1 root root 2177 Jan 9 2015 sam_init_virt_plugin.out.5443 -rw-rw-rw- 1 root root 2177 Jan 9 2015 sam_init_virt_plugin.out.5424 -rw-rw-rw- 1 root root 2177 Jan 9 2015 sam_init_virt_plugin.out.5423 -rw-rw-rw- 1 root root 2177 Jan 9 2015 sam_init_virt_plugin.out.5420 -rw-rw-rw- 1 root root 2362 Jan 9 2015 sam_init_virt_plugin.out.5356 -rw-rw-rw- 1 root root 2547 Jan 9 2015 sam_init_virt_plugin.out.5380 -rw-rw-rw- 1 root root 2706 Jan 9 2015 _clis_parse_tree_.log -rw-r--r-- 1 root root 2741 Jan 9 2015 bond_setup.log -rw-rw-rw- 1 root root 2883 Jun 13 10:44 sksd_crypt_service_1 -rw-rw-rw- 1 root root 4099 Jan 9 2015 stp.log.1 -rwxr-xr-x 1 root root 4285 Jun 11 10:37 iptab.sh -rw-rw-rw- 1 root root 5241 Jan 9 2015 first_setup.log -rw-rw-rw- 1 root root 8456 Dec 17 15:17 dave.log -rw-rw-rw- 1 root root 10748 Jun 11 07:55 snmpd-debug-sam.txt -rw-rw-rw- 1 root root 17607 Jan 9 2015 fcoe_mgr_init.log -rw-rw-rw- 1 root root 21356 Jan 9 2015 fm_debug.log.vdc_1.4927 -rw-rw-rw- 1 root root 62198 Jun 11 07:55 syslogd_errlog.4894_4894 -rw-rw-rw- 1 root root 174261 Jun 13 13:37 afm_srv_15.log -rw-r--r-- 1 root root 200394 Jan 9 2015 sam_bridge_boot.log -rw-rw-rw- 1 root root 250034 Jun 11 03:42 afm_srv_15.log.0 -rw-rw-rw- 1 root root 57569165 Jun 13 13:36 smm.log Linux(debug)# cp /dev/null /var/tmp/smm.log Linux(debug)# df -h Filesystem Size Used Avail Use% Mounted on /dev/root 300M 180M 121M 60% / none 2.0M 4.0K 2.0M 1% /post none 300M 180M 121M 60% /var none 3.0G 1008M 2.0G 34% /isan none 600M 1.1M 599M 1% /var/tmp none 2.0G 69M 1.9G 4% /var/sysmgr none 140M 72K 140M 1% /var/sysmgr/ftp none 128M 0 128M 0% /callhome none 512M 281M 232M 55% /dev/shm none 240M 77M 164M 32% /volatile none 8.0M 0 8.0M 0% /debug /dev/mtdblock2 48M 1.4M 47M 3% /mnt/plog /dev/sda5 84M 4.3M 75M 6% /mnt/cfg/0 /dev/sda6 84M 4.3M 75M 6% /mnt/cfg/1 none 400M 4.5M 396M 2% /var/sysmgr/startup-cfg /dev/sda4 84M 7.3M 72M 10% /mnt/pss /dev/sda3 16G 6.8G 8.4G 45% /bootflash /dev/sda7 3.8G 75M 3.6G 3% /opt /dev/mtdblock3 3.9M 2.2M 1.6M 58% /opt/db/nvram /dev/sda8 3.8G 35M 3.6G 1% /workspace /dev/sda9 5.7G 33M 5.3G 1% /spare tmpfs 1.0M 40K 984K 4% /bootflash/sysdebug/tftpd_logs none 50M 6.4M 44M 13% /debug-plugin Linux(debug)# exit exit
Check reboot issue in Firmware 2.2(3d) BUG63966
Another Bug we ran into some months ago 🙁
The problem is, a config save during shutdown or reboot maybe can’t happen because the system stops before the save is complete. -> Cisco’s resolution is to append a sleeptimer 😉
Here is how Cisco TAC Support fixed that:
UCS2-B# connect local-mgmt b
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
UCS2-B(local-mgmt)# cd debug_plugin/
UCS2-B(local-mgmt)# copy ucs-dplug.5.2.3.N2.2.22c.gbin x
UCS2-B(local-mgmt)# load x
###############################################################
Warning: debug-plugin is for engineering internal use only!
For security reason, plugin image has been deleted.
###############################################################
Successfully loaded debug-plugin!!!
Linux(debug)# cd /etc/init.d
Linux(debug)# ls
apachecracklibd dhcp functions getresvport.sh halt init-functions iptables killall netconsole netfsnetwork nfs-common nfs-user-server portmap portmap-wrapperracoonrc rdisc services-wrapper single syslog udev-post
Linux(debug)# cd /isan/bn in
Linux(debug)# cat sam_logs.sh
#!/bin/sh
/bin/mkdir -p /opt
cd /var/sysmgr
/bin/tar -czf /opt/sam_logs.tgz sam_logs
##switch logs
/bin/mkdir -p /bootflash/sw_trace_logs/
cp /var/sysmgr/tmp_logs/bcm_usd.log* /bootflash/sw_trace_logs/
cd /var/sysmgr/
/bin/tar -czf /bootflash/sw_trace_logs/sw_trace_logs_current.tgz sw_trace_logs/{*.current,*.1.gz} 2>/dev/null
/bin/tar -czf /bootflash/sw_trace_logs/sw_trace_logs_before_reboot.tgz sw_trace_logs 2>/dev/null
sync
sleep 10
Sure, you need to do this on both fabrics!
Check Backup
Upload the Bundles
Upload all your Bundles you downloaded from Cisco
Check Cluster Status
UCS-A# show cluster state Cluster Id: XXXXXXXXXXXXXXX A: UP, PRIMARY B: UP, SUBORDINATE
or GUI
Disable Home Call
Check out Part 2
One thought on “stumbling blocks in upgrading Cisco UCS (Part 1of2)”