UCS Upgrade failed

UCS Upgrade failed

The last days I tried to upgrade my UCS Domain from 2.2.7(c) to 2.2.8(g).

I have two UCS Domains. One of them went through the upgrade fine, the other one not :/

See what happend and how we fixed it …

I went through the steps I described in



In short form:

  1. Upgrade UCS Manager
  2. Upgrade IO Modules of the Chassis
  3. Upgrade Fabric Interconnect

After the reboot the upgraded Fabric Interconnect came up in Setup / Config Mode. This means there were no active config after the reboot.

WARNING: Please do the following with a Cisco TAC Engineer or at your own risk!


First we tried to re-connect to the running FI and get his config:

At this time the FI does not accept any commands, so we did a “hard” reboot. Again, there was no config on the FI.

We had to fix the boot partitions. During the boot process you have to constantly press “CRTL”+ “L”

Now we loaded the bootflash:

At this time we had to configure the IP of the FI:

To get the Debug Plugin on the switch, you have to get it from a TFTP Server

In the following step we unmounted the Filesystems and repaired them:

After two reboots the FI has it’s config

At this point the TAC Engineer told me he fixed this case.

Whooot???? Does not look like everything is fine! Ok, the FI has his config, but there are still errors.

Nerverless I had to open a new TAC Case to get this errors fixed:

[FSM:STAGE:RETRY:]: external VM manager extension-key configuration on local fabric(FSM-STAGE:sam:dme:ExtvmmMasterExtKeyConfig:SetPeer) F16898

[FSM:STAGE:REMOTE-ERROR]: Result: service-unavailable Code: unspecified Message: Error syncing extension key(sam:dme:ExtvmmMasterExtKeyConfig:SetPeer) F78338

[FSM:STAGE:REMOTE-ERROR]: Result: service-unavailable Code: unspecified Message: Error syncing extension key(sam:dme:ExtvmmProviderConfig:SetPeer) F78319

[FSM:STAGE:RETRY:]: external VM manager configuration on peer fabric(FSM-STAGE:sam:dme:ExtvmmProviderConfig:SetPeer) F16879

[FSM:FAILED]: external VM manager extension-key configuration(FSM:sam:dme:ExtvmmMasterExtKeyConfig). Remote-Invocation-Error: Error syncing extension key F999938

[FSM:FAILED]: external VM manager configuration(FSM:sam:dme:ExtvmmProviderConfig). Remote-Invocation-Error: Error syncing extension key F999919

At this time I was very disappointed with the TAC Support.


To got further I found the Bug CSCvf27661

So I opened a new TAC Case to exchange those SSH-Keys.

First to see the error:

Get the keys machting on both systems and edit it with vi šŸ˜‰

I cant say how wired the TAC Engineer does this, it took more than one hour to edit this two keys. But he also did a mistake! Be sure your key ends with “root@(none)”.

In my case the “(none)” was missing in one key. I changed it after the TAC Session and get all errors gone.


Hope this helps you …

Leave a comment and share!



Please also check CSCva31113. This explained my issue exactly.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.