stumbling blocks in upgrading Cisco UCS (Part 2of2)
Check Datapaths
It’s important to check at UCS AND your application (in my case VMware ESXi)
If all you have paths redundant, you can start updating …
Activate UCS Manager
At this point there is no impact for your paths. Its only the management instance.
But you will be logged out for one or two minutes. Log in again an check your Version.
Update IO Modules (Subordinate)
Now its important to grab your Subordinate IO Modules. Normally the IO Module 1 is connected to FI (A). But double check your Cables before 😉
BTW: Cross cabeling is not supported!
Select your IO Module and hit Update (Not Activate)
This took about < 10 minutes in my case.
Activate IO Modules (Subordinate)
When your update is complete, you can activate them now. Check the “Set Startup Version Only”
Activate FI (Subordinate)
This is the most critical step. Your multipathing have to work correctly now 😉
In my case I had some more troubles with this firmware 2.2(3d) BUG63966 because the FI reboots and comes up uncofigured. Cisco TAC Support fixed that in an 6 hour call ….
But back to normal upgrade.
You have to pick also your Subordinate FI and choose the lastest version available. Your FI reboots about 20-30 minutes. Be patient and no worrys about the alerts the you will see 😉
Here I lost my Fibrechannel link
After the upgrade of the FI, it came up, but the FC link didn’t.
You can see this failures in the MDS Log:
san-5a# sh logg | last 50 2016 Jun 13 09:31:57 san-5a %PORT-5-IF_DOWN_NONE: %$VSAN 3270%$ Interface fc1/11 is down (None) UCS2-A-P25 2016 Jun 13 09:31:57 san-5a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1/11 is down (OLS received) UCS2-A-P25 2016 Jun 13 09:32:36 san-5a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1/11 is down (OLS received) UCS2-A-P25 2016 Jun 13 09:32:36 san-5a %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1/11 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119) 2016 Jun 13 09:32:36 san-5a %PORT-5-IF_DOWN_NONE: %$VSAN 3270%$ Interface fc1/11 is down (None) UCS2-A-P25 2016 Jun 13 09:32:36 san-5a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1/11 is down (OLS received) UCS2-A-P25 2016 Jun 13 09:32:39 san-5a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1/11 is down (OLS received) UCS2-A-P25 2016 Jun 13 09:32:39 san-5a %PORT-5-IF_DOWN_OFFLINE: %$VSAN 3270%$ Interface fc1/11 is down (Offline) UCS2-A-P25 2016 Jun 13 09:33:19 san-5a %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1/11 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119) .....
I make it short, to fix this you should take care about the vsan trunking setting. Here your could see at UCS side its off.
And at MDS side, it should be off, too.
san-5a# conf t
Enter configuration commands, one per line. End with CNTL/Z.
san-5a(config)#
san-5a(config)# show interface fc1/11
fc1/11 is down (Initializing)
Port description is UCS2-A-P25
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:0b:00:05:73:ba:0b:98
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port vsan is 3270
Receive data field Size is 2112
Beacon is turned off
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
84698 frames input, 13252760 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
61796 frames output, 3027336 bytes
0 discards, 0 errors
4241 input OLS, 113 LRR, 38 NOS, 0 loop inits
5972 output OLS, 5857 LRR, 4138 NOS, 4126 loop inits
Interface last changed at Sat Jun 11 08:20:55 2016
san-5a(config)# interface fc1/11
san-5a(config-if)# switchport trunk mode off
san-5a(config-if)# shutdown
san-5a(config-if)# no shutdown
san-5a(config-if)# show interface brief
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/1 1000 auto on trunking swl TE 8 --
fc1/2 4 auto on up swl F 8 --
fc1/3 2 auto on up swl F 8 --
fc1/4 3270 auto on up swl F 8 --
fc1/5 3270 auto on up swl F 8 --
fc1/6 2 auto on up swl F 8 --
fc1/7 2 auto on up swl F 8 --
fc1/8 4 auto on up swl F 8 --
fc1/9 4 auto on up swl F 8 --
fc1/10 3270 auto on up swl F 8 --
fc1/11 3270 auto off up swl F 8 --
fc1/12 4 auto on up swl F 4 --
fc1/13 3270 auto on up swl FL 8 --
Again, Check Datapaths
It’s important to check at UCS AND your application (in my case VMware ESXi)
Halftime 😉
Failover FI
If you have all paths up and running, you should move the primary FI to subordinate.
UCS-A# show cluster state Cluster Id: 0xfc436fa8b88511e0-0xa370000573cb6c04 A: UP, SUBORDINATE B: UP, PRIMARY HA READY UCS-B# connect local-mgmt Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php UCS-B(local-mgmt)# cluster lead a UCS-B(local-mgmt)#
You will be disconnected if you have UCS Manager GUI open. Wait a minute an log in again.
Next steps are the same for the other Fabric:
- Update IO Modules (Now Subordinate)
- Activate IO Modules (Now Subordinate)
- Activate FI (Now Subordinate)
- Check Datapaths
Upgrade Servers/Blades
When you upgrade as discribed, don’t forget to upgrade your Servers too. (Firmware & Drivers)
Reward yourself
3 thoughts on “stumbling blocks in upgrading Cisco UCS (Part 2of2)”
“I couldn’t refrain from commenting. _Perfectly written!”
Thanks for reading!