stumbling blocks in upgrading Cisco UCS (Part 2of2)

stumbling blocks in upgrading Cisco UCS (Part 2of2)

Check Datapaths

It’s important to check at UCS AND your application (in my case VMware ESXi)

If all you have paths redundant, you can start updating …


Activate UCS Manager

At this point there is no impact for your paths. Its only the management instance.

But you will be logged out for one or two minutes. Log in again an check your Version.


Update IO Modules (Subordinate)

Now its important to grab your Subordinate IO Modules. Normally the IO Module 1 is connected to FI (A). But double check your Cables before 😉

BTW: Cross cabeling is not supported!

Select your IO Module and hit Update (Not Activate)

This took about < 10 minutes in my case.


Activate IO Modules (Subordinate)

When your update is complete, you can activate them now. Check the “Set Startup Version Only”


Activate FI (Subordinate)

This is the most critical step. Your multipathing have to work correctly now 😉

In my case I had some more troubles with this firmware  2.2(3d) BUG63966 because the FI reboots and comes up uncofigured. Cisco TAC Support fixed that in an 6 hour call ….

But back to normal upgrade.

You have to pick also your Subordinate FI and choose the lastest version available. Your FI reboots about 20-30 minutes. Be patient and no worrys about the alerts the you will see 😉


Here I lost my Fibrechannel link

After the upgrade of the FI, it came up, but the FC link didn’t.

You can see this failures in the MDS Log:

san-5a# sh logg | last 50
2016 Jun 13 09:31:57 san-5a %PORT-5-IF_DOWN_NONE: %$VSAN 3270%$ Interface fc1/11 is down (None)  UCS2-A-P25       
2016 Jun 13 09:31:57 san-5a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1/11 is down (OLS received)  UCS2-A-P25       
2016 Jun 13 09:32:36 san-5a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1/11 is down (OLS received)  UCS2-A-P25       
2016 Jun 13 09:32:36 san-5a %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1/11 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119)
2016 Jun 13 09:32:36 san-5a %PORT-5-IF_DOWN_NONE: %$VSAN 3270%$ Interface fc1/11 is down (None)  UCS2-A-P25       
2016 Jun 13 09:32:36 san-5a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1/11 is down (OLS received)  UCS2-A-P25       
2016 Jun 13 09:32:39 san-5a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1/11 is down (OLS received)  UCS2-A-P25       
2016 Jun 13 09:32:39 san-5a %PORT-5-IF_DOWN_OFFLINE: %$VSAN 3270%$ Interface fc1/11 is down (Offline)  UCS2-A-P25       
2016 Jun 13 09:33:19 san-5a %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1/11 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119)
.....

I make it short, to fix this you should take care about the vsan trunking setting. Here your could see at UCS side its off.

And at MDS side, it should be off, too.

san-5a# conf t

Enter configuration commands, one per line.  End with CNTL/Z.

san-5a(config)# 
san-5a(config)# show interface fc1/11

fc1/11 is down (Initializing)
    Port description is UCS2-A-P25
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is 20:0b:00:05:73:ba:0b:98
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port vsan is 3270
    Receive data field Size is 2112
    Beacon is turned off
    5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      84698 frames input, 13252760 bytes
        0 discards, 0 errors
        0 CRC,  0 unknown class
        0 too long, 0 too short
      61796 frames output, 3027336 bytes
        0 discards, 0 errors
      4241 input OLS, 113 LRR, 38 NOS, 0 loop inits
      5972 output OLS, 5857 LRR, 4138 NOS, 4126 loop inits
    Interface last changed at Sat Jun 11 08:20:55 2016
san-5a(config)# interface fc1/11
san-5a(config-if)# switchport trunk mode off
san-5a(config-if)# shutdown
san-5a(config-if)# no shutdown
san-5a(config-if)# show interface brief
-------------------------------------------------------------------------------
Interface  Vsan   Admin  Admin   Status          SFP    Oper  Oper   Port
                  Mode   Trunk                          Mode  Speed  Channel
                         Mode                                 (Gbps)
-------------------------------------------------------------------------------
fc1/1      1000   auto   on      trunking         swl    TE      8    --
fc1/2      4      auto   on      up               swl    F       8    --
fc1/3      2      auto   on      up               swl    F       8    --
fc1/4      3270   auto   on      up               swl    F       8    --
fc1/5      3270   auto   on      up               swl    F       8    --
fc1/6      2      auto   on      up               swl    F       8    --
fc1/7      2      auto   on      up               swl    F       8    --
fc1/8      4      auto   on      up               swl    F       8    --
fc1/9      4      auto   on      up               swl    F       8    --
fc1/10     3270   auto   on      up               swl    F       8    --
fc1/11     3270   auto   off     up               swl    F       8    --
fc1/12     4      auto   on      up               swl    F       4    --
fc1/13     3270   auto   on      up               swl    FL      8    --

Again, Check Datapaths

It’s important to check at UCS AND your application (in my case VMware ESXi)


Halftime 😉


Failover FI

If you have all paths up and running, you should move the primary FI to subordinate.

UCS-A# show cluster state
Cluster Id: 0xfc436fa8b88511e0-0xa370000573cb6c04

A: UP, SUBORDINATE
B: UP, PRIMARY

HA READY
UCS-B# connect local-mgmt
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

UCS-B(local-mgmt)# cluster lead a
UCS-B(local-mgmt)#

You will be disconnected if you have UCS Manager GUI open. Wait a minute an log in again.


Next steps are the same for the other Fabric:

  • Update IO Modules (Now Subordinate)
  • Activate IO Modules (Now Subordinate)
  • Activate FI (Now Subordinate)
  • Check Datapaths

Upgrade Servers/Blades

When you upgrade as discribed, don’t forget to upgrade your Servers too. (Firmware & Drivers)


Reward yourself

 

3 thoughts on “stumbling blocks in upgrading Cisco UCS (Part 2of2)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.