{"id":1027,"date":"2021-10-28T12:46:28","date_gmt":"2021-10-28T10:46:28","guid":{"rendered":"http:\/\/172.16.10.10:8080\/?p=1027"},"modified":"2023-03-29T11:28:45","modified_gmt":"2023-03-29T09:28:45","slug":"cisco-ucs-and-mds-fibrechannel-ols-errors","status":"publish","type":"post","link":"http:\/\/vblog.hochsticher.de\/?p=1027","title":{"rendered":"Cisco UCS and MDS FibreChannel OLS Errors"},"content":{"rendered":"\n

Recently we installed a brand new UCS System with 6454 Fabric Interconnects attached to Cisco MDS Fibre Channel Switches. But we get into some trouble with FirbeChannel Ports and their PortChannels.<\/p>\n\n\n\n\n\n\n\n

But from the beginning \u2026<\/p>\n\n\n\n

Our Setup was the following:
UCS FI 6454 Version 4.2.1f
Cisco 9148 MDS Version 6.2.33
Cisco 9396S MDS Version 8.4(2)
Cisco B200 M5 Blades<\/p>\n\n\n\n

I created a PortChannel between the new UCS and the MDS. PortChannel came up as expected. Everything looked fine at this time. So I went further an installed the Blades with VMware ESXi 6.7 U3 in my AutoDeply Setup. Also fine at this point. So I created the new FC zoning to my Huawei Storage.<\/p>\n\n\n\n

The “big bang” came when I did a “Rescan Adapter” on the ESXi Host. I got Interface Errors and even the complete PortChannel went down.<\/p>\n\n\n\n

2021 Sep  1 11:58:46 san-2a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1\/4 is down (OLS received) port-channel2 ucs3\n2021 Sep  1 11:58:46 san-2a %PORT-CHANNEL-5-PORT_DOWN: port-channel2: fc1\/4 is down\n2021 Sep  1 12:01:14 san-2a %PORT-5-IF_DOWN_NONE: %$VSAN 3270%$ Interface port-channel2 is down (None)  ucs3\n2021 Sep  1 12:01:14 san-2a %PORT-CHANNEL-5-FOP_CHANGED: port-channel2: first operational port changed from fc1\/3 to none\n2021 Sep  1 12:01:14 san-2a %PORT-CHANNEL-5-PORT_DOWN: port-channel2: fc1\/3 is down\n2021 Sep  1 12:01:14 san-2a %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: %$VSAN 3270%$ Interface port-channel2 is down (No operational members)  ucs3\n2021 Sep  1 12:01:15 san-2a %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1\/3 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS\/LOS) (0x119)\n2021 Sep  1 12:01:15 san-2a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1\/3 is down (OLS received) port-channel2 ucs3\n\n<\/code><\/pre>\n\n\n\n
For troubleshooting purposes I went from PortChannel to a single Interface:<\/p>\n\n\n\n
2021 Sep  2 09:17:04 san-2a %PORT-CHANNEL-5-DELETED: port-channel2 deleted\n2021 Sep  2 09:18:53 san-2a %PORT-5-IF_UP: %$VSAN 3270%$ Interface fc1\/3 is up in mode F  ucs3\n2021 Sep  2 09:32:04 san-2a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1\/3 is down (OLS received)  ucs3\n2021 Sep  2 09:32:05 san-2a %PORT-5-IF_UP: %$VSAN 3270%$ Interface fc1\/3 is up in mode F  ucs3\n2021 Sep  2 09:36:32 san-2a %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3270%$ Interface fc1\/3 is down (OLS received)  ucs3\n2021 Sep  2 09:36:33 san-2a %PORT-5-IF_UP: %$VSAN 3270%$ Interface fc1\/3 is up in mode F  ucs3\n<\/code><\/pre>\n\n\n\n
But I got the same Errors :\/<\/p>\n\n\n\n
Next try was a newer Cisco 9396S MDS Switch:<\/p>\n\n\n\n
2021 Sep 3 08:41:10 san-1b %PORT-5-IF_DOWN_NONE: %$VSAN 3370%$ Interface port-channel32 is down (None)\n2021 Sep 3 08:41:10 san-1b %PORT-CHANNEL-5-FOP_CHANGED: port-channel32: first operational port changed from fc1\/11 to none\n2021 Sep 3 08:41:10 san-1b %PORT-CHANNEL-5-PORT_DOWN: port-channel32: fc1\/11 is down\n2021 Sep 3 08:41:10 san-1b %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: %$VSAN 3370%$ Interface port-channel32 is down (No operational members)\n2021 Sep 3 08:41:10 san-1b %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1\/11 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS\/LOS) (0x119)\n2021 Sep 3 08:41:10 san-1b %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3370%$ Interface fc1\/11 is down (OLS received) port-channel32\n2021 Sep 3 08:41:40 san-1b %PORT-5-IF_DOWN: %$VSAN 3370%$ Interface fc1\/11 is down (Gracefully shutdown) port-channel32\n2021 Sep 3 08:41:40 san-1b %PORT-CHANNEL-5-DELETED: port-channel32 deleted\n2021 Sep 3 08:42:42 san-1b %LIBIFMGR-5-INTF_COUNTERS_CLEARED: Interface fc1\/11, counters cleared by user\n2021 Sep 3 08:42:58 san-1b %PORT-5-IF_UP: %$VSAN 3370%$ Interface fc1\/11 is up in mode F\n2021 Sep 3 08:46:22 san-1b %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 3370%$ Interface fc1\/11 is down (OLS received)\n2021 Sep 3 08:46:23 san-1b %PORT-5-IF_UP: %$VSAN 3370%$ Interface fc1\/11 is up in mode F\n2021 Sep 3 08:47:42 san-1b %PMON-SLOT1-3-RISING_THRESHOLD_REACHED: TX Credit Not Available has reached the rising threshold (port=fc1\/11 [0x100a000], value=20) .\n2021 Sep 3 08:47:43 san-1b %PMON-SLOT1-3-RISING_THRESHOLD_REACHED: Credit Loss Reco has reached the rising threshold (port=fc1\/11 [0x100a000], value=1) .\n2021 Sep 3 08:47:44 san-1b %PMON-SLOT1-3-FALLING_THRESHOLD_REACHED: Credit Loss Reco has reached the falling threshold (port=fc1\/11 [0x100a000], value=0) .\n2021 Sep 3 08:47:44 san-1b %PMON-SLOT1-3-FALLING_THRESHOLD_REACHED: TX Credit Not Available has reached the falling threshold (port=fc1\/11 [0x100a000], value=0) .\n2021 Sep 3 08:52:28 san-1b %PMON-SLOT1-3-RISING_THRESHOLD_REACHED: TX Credit Not Available has reached the rising threshold (port=fc1\/11 [0x100a000], value=10) .\n2021 Sep 3 08:52:29 san-1b %PMON-SLOT1-3-RISING_THRESHOLD_REACHED: Credit Loss Reco has reached the rising threshold (port=fc1\/11 [0x100a000], value=1) .\n2021 Sep 3 08:52:31 san-1b %PMON-SLOT1-3-FALLING_THRESHOLD_REACHED: Credit Loss Reco has reached the falling threshold (port=fc1\/11 [0x100a000], value=0) .\n2021 Sep 3 08:52:31 san-1b %PMON-SLOT1-3-FALLING_THRESHOLD_REACHED: TX Credit Not Available has reached the falling threshold (port=fc1\/11 [0x100a000], value=0) .\n<\/code><\/pre>\n\n\n\n
Same OLS Errors and some more. But still no stable FirbeChannel connection. The Interfaces always had this errors:<\/p>\n\n\n\n
3 input OLS,1  LRR,0 NOS,0 loop inits\n2 output OLS,1 LRR, 2 NOS, 2 loop inits\n<\/code><\/pre>\n\n\n\n
Of course I did some research and found the following:<\/p>\n\n\n\n
https:\/\/bst.cloudapps.cisco.com\/bugsearch\/bug\/CSCvn14381\/?rfs=iqvred
<\/a>-> Should not hit because Firmware is higher.
https:\/\/bst.cloudapps.cisco.com\/bugsearch\/bug\/CSCvo08627\/?rfs=iqvred
<\/a>-> Unsure what the solution is.
https:\/\/kb.netapp.com\/Advice_and_Troubleshooting\/Data_Storage_Systems\/Fabric%2C_Interconnect_and_Management_Switches\/Cisco_MDS_switch_reporting_%22TX_Credit_Not_Available_has_reached_the_rising_threshold%22<\/a>
-> We changed enough cable and SFPs.
https:\/\/www.dell.com\/support\/kbdoc\/de-ch\/000167985\/connectrix-mds-port-channel-between-cisco-mds-and-ucs-not-working
<\/a>-> Single uplinks already tried.
https:\/\/bst.cloudapps.cisco.com\/bugsearch\/bug\/CSCvv84472
<\/a>-> the FI 6454 has the 8c604f as oui and is included on the 9148 and 9396S. Also single uplink should work.<\/p>\n\n\n\n
But none of them helped me out.<\/p>\n\n\n\n
<\/p>\n\n\n\n
Frustrated I went to UCS GUI and tripple checked my settings. But there is not much to configure. But in single uplink mode I found a setting about the Fill Pattern which cannot be changed and is set to “Idle”.<\/p>\n\n\n\n
\n
\"\"<\/figure>\n<\/figure>\n\n\n\n
I looked in my old UCS System 6248 the Fill Pattern can be changed and the default is set to “Arbff”<\/p>\n\n\n\n
\n
\"\"<\/figure>\n<\/figure>\n\n\n\n
I did some research about the “Fill Word” and found the following:<\/p>\n\n\n\n
\u201eCisco UCS 6400 Series Fabric Interconnects do not support 8 Gbps direct-attached FC connectivity (FC uplink ports or FC storage ports) without fill-pattern set to IDLE. When migrating to Cisco UCS 6400 Series Fabric Interconnects from Cisco UCS 6200 Series Fabric Interconnects, do one of the following:\u201c<\/p>\n\n\n\n
                \u2022 Use a SAN switch between the Cisco UCS 6400 Series Fabric Interconnect and the storage array with 8 GB FC connectivity.
                \u2022 Upgrade the storage array to 16 GB or 32 GB FC connectivity.<\/p>\n\n\n\n
https:\/\/www.cisco.com\/c\/en\/us\/td\/docs\/unified_computing\/ucs\/ucs-manager\/GUI-User-Guides\/Storage-Mgmt\/4-2\/b_UCSM_GUI_Storage_Management_Guide_4_2\/b_UCSM_GUI_Storage_Management_Guide_chapter_01110.html<\/a><\/p>\n\n\n\n
<\/p>\n\n\n\n
So that means you can’t change this setting on the UCS, but on the MDS. I went to my MDS Console and tried:<\/p>\n\n\n\n
san-2a(config-if)# switchport fill-pattern IDLE speed 8000<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/div>\n\n\n\n
<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n\n\n\n
Since that change all my Ports stayed stable and also later in PortChannel configuration \ud83d\ude09<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"
Recently we installed a brand new UCS System with 6454 Fabric Interconnects attached to Cisco MDS Fibre Channel Switches. But we get into some trouble with FirbeChannel Ports and their PortChannels.<\/p>\n","protected":false},"author":1,"featured_media":1069,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[2,3,7],"tags":[],"class_list":["post-1027","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cisco-mds","category-cisco-ucs","category-vsphere"],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/posts\/1027","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1027"}],"version-history":[{"count":14,"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/posts\/1027\/revisions"}],"predecessor-version":[{"id":1100,"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/posts\/1027\/revisions\/1100"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/media\/1069"}],"wp:attachment":[{"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1027"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1027"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/vblog.hochsticher.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1027"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}