{"id":862,"date":"2017-11-03T10:44:52","date_gmt":"2017-11-03T09:44:52","guid":{"rendered":"https:\/\/vblog.hochsticher.de\/?p=862"},"modified":"2023-03-30T10:33:47","modified_gmt":"2023-03-30T08:33:47","slug":"ucs-upgrade-failed","status":"publish","type":"post","link":"https:\/\/vblog.hochsticher.de\/?p=862","title":{"rendered":"UCS Upgrade failed"},"content":{"rendered":"

The last days I tried to upgrade my UCS Domain from 2.2.7(c) to 2.2.8(g).<\/p>\n

I have two UCS Domains. One of them went through the upgrade fine, the other one not :\/<\/p>\n

See what happend and how we fixed it …<\/p>\n

<\/p>\n

I went through the steps I described in<\/p>\n

STUMBLING BLOCKS IN UPGRADING CISCO UCS (PART 1OF2)<\/a><\/p>\n

STUMBLING BLOCKS IN UPGRADING CISCO UCS (PART 2OF2)<\/a><\/p>\n

In short form:<\/p>\n

    \n
  1. Upgrade UCS Manager<\/li>\n
  2. Upgrade IO Modules of the Chassis<\/li>\n
  3. Upgrade Fabric Interconnect<\/li>\n<\/ol>\n

    After the reboot the upgraded Fabric Interconnect came up in Setup \/ Config Mode. This means there were no active config after the reboot.<\/p>\n

    \"\"<\/p>\n

    WARNING: Please do the following with a Cisco TAC Engineer or at your own risk!<\/em><\/strong><\/p>\n

     <\/p>\n

    First we tried to re-connect to the running FI and get his config:<\/p>\n

    Type the hot key to suspend the connection: <CTRL>Q\n\n  Enter the configuration method. (console\/gui) ?     console\n\n  Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the cluster. Continue (y\/n) ? y\n\n  Enter the admin password of the peer Fabric interconnect: \n    Connecting to peer Fabric interconnect... unable to connect! Password could be wrong.\n    Please ensure that the authentication mode on peer Fabric interconnect is set to 'Local'\n    Hit enter to try again or type 'restart' to start setup from beginning... \n    ? \n\n   Connecting to peer Fabric interconnect... done\n    Retrieving config from peer Fabric interconnect... done\n\/isan\/bin\/getversion: error while loading shared libraries: libosiris.so: cannot open shared object file: No such file or directory\n    Installer has determined that the peer Fabric Interconnect is running a different firmware version than the local Fabric. Cannot join cluster.\n \n    Local Fabric Interconnect\n      UCSM version     : \n      Kernel version   : \n      System version   : \n      local_model_no   : 6248\n\n    Peer Fabric Interconnect\n      UCSM version     : 2.2(8g)\n      Kernel version   : 5.2(3)N2(2.27c)\n      System version   : 5.2(3)N2(2.27c)\n      peer_model_no    : 6248\n\n\n  Do you wish to update firmware on this Fabric Interconnect to the Peer's version? (y\/n): y\nUpdating firmware of Fabric Interconnect....... [ Please don't press Ctrl+c while updating firmware ]\n Updating images \n Please wait for firmware update to complete.... \n Checking the Compatibility of new Firmware..... [ Please don't Press ctrl+c ]. \nVerifying image bootflash:\/installables\/switch\/ucs-6100-k9-kickstart.5.2.3.N2.2.27c.bin for boot variable \"kickstart\".\n[#                   ]   0%[####################] 100% -- SUCCESS\n\nVerifying image bootflash:\/installables\/switch\/ucs-6100-k9-system.5.2.3.N2.2.27c.bin for boot variable \"system\".\n[#                   ]   0%[####################] 100% -- SUCCESS\n\nVerifying image type.\n[#                   ]   0%[#####               ]  20%[#######             ]  30%[#########           ]  40%[###########         ]  50%[###########         ]  50%[###########         ]  50%[################### ]  90%[####################] 100%[####################] 100% -- SUCCESS\n\nExtracting \"system\" version from image bootflash:\/installables\/switch\/ucs-6100-k9-system.5.2.3.N2.2.27c.bin.\n[#                   ]   0%[####################] 100% -- SUCCESS\n\nExtracting \"kickstart\" version from image bootflash:\/installables\/switch\/ucs-6100-k9-kickstart.5.2.3.N2.2.27c.bin.\n[#                   ]   0%[####################] 100% -- SUCCESS\n\nExtracting \"bios\" version from image bootflash:\/installables\/switch\/ucs-6100-k9-system.5.2.3.N2.2.27c.bin.\n[#                   ]   0%[####################] 100% -- SUCCESS\n\nPerforming module support checks.\n[####################] 100% -- SUCCESS\n\nNotifying services about system upgrade.\n[####################] 100% -- SUCCESS\n\n\n\nCompatibility check is done:\nModule  bootable          Impact  Install-type  Reason\n------  --------  --------------  ------------  ------\n     1       yes      disruptive         reset  Incompatible image\n\n\n\nImages will be upgraded according to following table:\nModule       Image         Running-Version             New-Version  Upg-Required\n------  ----------  ----------------------  ----------------------  ------------\n     1      system         5.2(3)N2(2.28g)         5.2(3)N2(2.27c)           yes\n     1   kickstart         5.2(3)N2(2.28g)         5.2(3)N2(2.27c)           yes\n     1        bios      v3.6.0(05\/09\/2012)      v3.6.0(05\/09\/2012)            no\n     1      SFP-uC                v1.1.0.0                v1.0.0.0            no\n     1   power-seq                    v3.0                    v3.0            no\n     3   power-seq                    v2.0                    v2.0            no\n     1          uC                v1.2.0.1                v1.2.0.1            no\n\n\nSwitch will be reloaded for disruptive upgrade.\n\nInstall is in progress, please wait.\n\nPerforming runtime checks.\n[####################] 100% -- SUCCESS\n\nSetting boot variables.\n[#                   ]   0%[####################] 100% -- SUCCESS\n\nPerforming configuration copy.\n[#                   ]   0%[###                 ]  10%[####                ]  15%[#####               ]  20%[######              ]  25%[#######             ]  30%[########            ]  35%[#########           ]  40%[##########          ]  45%[###########         ]  50%[#############       ]  60%[##############      ]  65%[###############     ]  70%[################    ]  75%[#################   ]  80%[##################  ]  85%[################### ]  90%[####################]  95%[####################] 100%[####################] 100% -- SUCCESS\n\nConverting startup config.\n[#                   ]   0%[####################] 100% -- SUCCESS\n\nInstall has been successful.\n Firmware Updation Successfully Completed. Please wait to enter the IP address \n\n\n\n\n\n  Type 'reboot' to abort configuration and reboot system\n  or hit enter to continue. (reboot\/<CR>) ? \n    Peer Fabric interconnect Mgmt0 IPv4 Address: 10.0.0.1\n    Peer Fabric interconnect Mgmt0 IPv4 Netmask: 255.255.255.0\n    Cluster IPv4 address          : 10.0.0.3\n \n    Peer FI is IPv4 Cluster enabled. Please Provide Local Fabric Interconnect Mgmt0 IPv4 Address  \n\n  Physical Switch Mgmt0 IP address : \n\n   Mgmt0 IP must be specified\n\n  Physical Switch Mgmt0 IP address : 10.0.0.2\n\n\n  Apply and save the configuration (select 'no' if you want to re-enter)? (yes\/no): \n  Type 'reboot' to abort configuration and reboot system\n  or hit enter to continue. (reboot\/<CR>) ? \n\n\n  Apply and save the configuration (select 'no' if you want to re-enter)? (yes\/no): yes\n  Applying configuration. Please wait.\n\nTue Oct 31 11:59:53 UTC 2017\n\n  Type 'reboot' to abort configuration and reboot system\n  or hit enter to continue. (reboot\/<CR>) ?   Configuration file - Ok\n2017 Oct 31 12:00:10 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256\n\n2017 Oct 31 12:00:10 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:00:11 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256\n\n2017 Oct 31 12:00:11 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:00:11 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256\n\n2017 Oct 31 12:00:11 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:00:11 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256\n\n2017 Oct 31 12:00:11 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:00:12 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256\n\n2017 Oct 31 12:00:12 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:00:12 UCS1-B %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: dhcpd - pmon\n\n\n\nUser Access Verification\nUCS1-B login: admin\nPassword: \nCisco Nexus Operating System (NX-OS) Software\nTAC support: http:\/\/www.cisco.com\/tac\nCopyright (c) 2002-2017, Cisco Systems, Inc. All rights reserved.\nThe copyrights to certain works contained in this software are\nowned by other third parties and used and distributed under\nlicense. Certain components of this software are licensed under\nthe GNU General Public License (GPL) version 2.0 or the GNU\nLesser General Public License (LGPL) Version 2.1. A copy of each\nsuch license is available at\nhttp:\/\/www.opensource.org\/licenses\/gpl-2.0.php and\nhttp:\/\/www.opensource.org\/licenses\/lgpl-2.1.php<\/pre>\n
    At this time the FI does not accept any commands, so we did a “hard” reboot. Again, there was no config on the FI.<\/p>\n
    N5000 BIOS v.3.6.0, Wed 05\/09\/2012, 03:15 PM \n\n989CB4B4B4B4B4B4B4B4B4B49999999299A0A2A3A0A2A3B2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             B2Version 2.00.1201. Copyright (C) 2009 American Megatrends, Inc.                 Booting kickstart image: bootflash:\/installables\/switch\/ucs-6100-k9-kickstart.5\n.2.3.N2.2.27c.bin....\n...............................................................................\n...........................................Image verification OK\n\nUsage: init 0123456SsQqAaBbCcUu\n\nINIT: [   10.657597] I2C - Mezz absent \nStarting system POST.....\n  Executing Mod 1 1 SEEPROM Test:...done (0 seconds)\n  Executing Mod 1 1 GigE Port Test:....done (32 seconds)\n  Executing Mod 1 1 PCIE Test:.................done (0 seconds)\n  Mod 1 1 Post Completed Successfully\nPOST is completed\ncan't create lock file \/var\/lock\/mtab~207: No such file or directory (use -n flag to override)\nS10mount-ramfs.supnuovaca Mounting \/isan 3000m\nMounted \/isan\nCreating \/callhome..\nMounting \/callhome..\nCreating \/callhome done.\nCallhome spool file system init done.\nnohup: redirecting stderr to stdout\nautoneg unmodified, ignoring\nautoneg unmodified, ignoring\nChecking all filesystems..r.r.r. done.\nChecking NVRAM block device ... done\nThe startup-config won't be used until the next reboot.\n. \nLoading system software\nUncompressing system image: bootflash:\/installables\/switch\/ucs-6100-k9-system.5.2.3.N2.2.27c.bin\n\nLoading plugin 0: core_plugin...\nLoading plugin 1: eth_plugin...\nLoading plugin 2: fc_plugin...\n\n\n13+1 records in\n13+1 records out\n10240 bytes (10 kB) copied, 5.7017e-05 s, 180 MB\/s\nethernet end-host mode on CA\nFC end-host mode on CA\nn_port virtualizer mode.\n---------------------------------------------------------------\n\nINIT: Entering runlevel: 3\n\ntouch: cannot touch `\/var\/lock\/subsys\/netfs': No such file or directory\n\/isan\/bin\/muxif_config: fex vlan id: -f,4042\nSet name-type for VLAN subsystem. Should be visible in \/proc\/net\/vlan\/config\nAdded VLAN with VID == 4042 to IF -:muxif:-\ncp: cannot stat `\/isan\/plugin_img\/fex.bin': No such file or directory\n\n---------------------\nenabled fc feature\n---------------------\n2017 Oct 31 12:09:31  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: loading cmd files begin  - clis\n\n2017 Oct 31 12:09:34  %$ VDC-1 %$ Oct 31 12:09:34 %KERN-0-SYSTEM_MSG: [   10.657597] I2C - Mezz absent  - kernel\n\n2017 Oct 31 12:09:41  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: loading cmd files end  - clis\n\n2017 Oct 31 12:09:41  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: init begin  - clis\n\n2017 Oct 31 12:09:49  %$ VDC-1 %$ %SNMPD-2-CRITICAL: SNMP log critical : load_mib_module :Error, while loading the mib module \/isan\/lib\/libsvc_sam_extSnmpPlugin.so (\/isan\/lib\/libsvc_sam_extSnmpPlugin.so: cannot open shared object file: No such file or directory)  \n\n2017 Oct 31 12:09:54  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_controller crashed with crash type:32512\n\n2017 Oct 31 12:09:54  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:54  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:32512\n\n2017 Oct 31 12:09:54  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:54  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:32512\n\n2017 Oct 31 12:09:54  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:55  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:32512\n\n2017 Oct 31 12:09:55  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:55  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:32512\n\n2017 Oct 31 12:09:55  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:55  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:32512\n\n2017 Oct 31 12:09:55  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:56  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:32512\n\n2017 Oct 31 12:09:56  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:56  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:32512\n\n2017 Oct 31 12:09:56  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:56  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:32256\n\n2017 Oct 31 12:09:56  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:56  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:32512\n\n2017 Oct 31 12:09:56  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:57  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_controller crashed with crash type:32512\n\n2017 Oct 31 12:09:57  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:57  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:32512\n\n2017 Oct 31 12:09:57  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:57  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:32512\n\n2017 Oct 31 12:09:57  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:58  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:32512\n\n2017 Oct 31 12:09:58  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:58  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:32512\n\n2017 Oct 31 12:09:58  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:09:59  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:32512\n\n2017 Oct 31 12:09:59  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:00  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:32512\n\n2017 Oct 31 12:10:00  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:00  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:32512\n\n2017 Oct 31 12:10:00  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:01  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:32256\n\n2017 Oct 31 12:10:01  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:01  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:32512\n\n2017 Oct 31 12:10:01  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:02  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_controller crashed with crash type:32512\n\n2017 Oct 31 12:10:02  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:03  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:32512\n\n2017 Oct 31 12:10:03  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:03  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:32512\n\n2017 Oct 31 12:10:03  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:04  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:32512\n\n2017 Oct 31 12:10:04  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:04  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:32512\n\n2017 Oct 31 12:10:04  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:04  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:32512\n\n2017 Oct 31 12:10:04  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:05  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:32512\n\n2017 Oct 31 12:10:05  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:05  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:32512\n\n2017 Oct 31 12:10:05  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:05  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:32256\n\n2017 Oct 31 12:10:05  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:06  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:32512\n\n2017 Oct 31 12:10:06  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:06  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_controller crashed with crash type:32512\n\n2017 Oct 31 12:10:06  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:06  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_controller - pmon\n\n2017 Oct 31 12:10:06  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:32512\n\n2017 Oct 31 12:10:06  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:06  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_dme - pmon\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:32512\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_dcosAG - pmon\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:32512\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_bladeAG - pmon\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:32512\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:07  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_portAG - pmon\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:32512\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_hostagentAG - pmon\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:32512\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_nicAG - pmon\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:32512\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:08  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_extvmmAG - pmon\n\n2017 Oct 31 12:10:09  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:32256\n\n2017 Oct 31 12:10:09  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:09  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_cliD - pmon\n\n2017 Oct 31 12:10:09  %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:32512\n\n2017 Oct 31 12:10:09  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n2017 Oct 31 12:10:09  %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_pamProxy - pmon\n\n2017 Oct 31 12:10:27  %$ VDC-1 %$ %CALLHOME-2-EVENT: httpd.sh crashed with crash type:0\n\n2017 Oct 31 12:10:27  %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\nSystem is coming up ... Please wait ...\nSystem is coming up ... Please wait ...\nSystem is coming up ... Please wait ...\n2017 Oct 31 12:10:46  %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online \n\nSystem is coming up ... Please wait ...\nnohup: appending output to `nohup.out'\n2017 Oct 31 12:11:01 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Running in PIO stats mode  - carmelusd\n\n2017 Oct 31 12:11:01 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: httpd.sh crashed with crash type:0\n\n2017 Oct 31 12:11:01 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n\n\n           ---- Basic System Configuration Dialog ----\n\n  This setup utility will guide you through the basic configuration of\n  the system. Only minimal configuration including IP connectivity to\n  the Fabric interconnect and its clustering mode is performed through these steps.\n\n  Type Ctrl-C at any time to abort configuration and reboot system.\n  To back track or make modifications to already entered values,\n  complete input till end of section and answer no when prompted\n  to apply configuration.\n\n  \n  Enter the configuration method. (console\/gui) ? \n  Type 'reboot' to abort configuration and reboot system\n  or hit enter to continue. (reboot\/<CR>) ? 2017 Oct 31 12:11:34 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: httpd.sh crashed with crash type:0\n\n2017 Oct 31 12:11:34 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH\n<\/pre>\n
    We had to fix the boot partitions. During the boot process you have to constantly press “CRTL”+ “L”<\/strong><\/p>\n
    N5000 BIOS v.3.6.0, Wed 05\/09\/2012, 03:15 PM \n\n989CB4B4B4B4B4B4B4B4B4B49999999299A0A2A3A0A2A3B2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             B2Version 2.00.1201. Copyright (C) 2009 American Megatrends, Inc.                 \nUser break into bootloader\n\n\nloader>                                                                        dir\nbootflash:\n  span.log\n  ucs-6100-k9-kickstart.5.0.3.N2.2.02q.bin\n  ucs-6100-k9-system.5.0.3.N2.2.02q.bin\n  chassis.img\n  pnuos\n  nuova-sim-mgmt-nsg.0.1.0.001.bin\n  chassis2.img\n  fexth.bin\n  installables\n  sysdebug\n  distributables_hdr\n  cores\n  techsupport\n  mts.log\n  vdc_2\n  vdc_3\n  vdc_4\n  distributables\n  initial_setup.log\n  license\n  received<\/pre>\n
    Now we loaded the bootflash:<\/p>\n
    loader>                                                                         boot bootflash:\/installables\/switch\/ucs-6100-k9-kickstart.5.2.3.N2.2\n<.2.3.N2.2.                                                                    27c.bin\nBooting kickstart image: bootflash:\/installables\/switch\/ucs-6100-k9-kickstart.5\n.2.3.N2.2.27c.bin....\n...............................................................................\n...........................................Image verification OK\nUsage: init 0123456SsQqAaBbCcUu\n INIT: [ 10.668160] I2C - Mezz absent \nStarting system POST.....\n Executing Mod 1 1 SEEPROM Test:...done (0 seconds)\n Executing Mod 1 1 GigE Port Test:....done (32 seconds)\n Executing Mod 1 1 PCIE Test:.................done (0 seconds)\n Mod 1 1 Post Completed Successfully\nPOST is completed\ncan't create lock file \/var\/lock\/mtab~207: No such file or directory (use -n flag to override)\nS10mount-ramfs.supnuovaca Mounting \/isan 3000m\nMounted \/isan\nCreating \/callhome..\nMounting \/callhome..\nCreating \/callhome done.\nCallhome spool file system init done.\nnohup: redirecting stderr to stdout\nautoneg unmodified, ignoring\nautoneg unmodified, ignoring\nChecking all filesystems..... done.\nChecking NVRAM block device ... done\nThe startup-config won't be used until the next reboot.\n. \nLoading system software\nNo system image Cisco Nexus Operating System (NX-OS) Software\nTAC support: http:\/\/www.cisco.com\/tac\nCopyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.\nThe copyrights to certain works contained in this software are\nowned by other third parties and used and distributed under\nlicense. Certain components of this software are licensed under\nthe GNU General Public License (GPL) version 2.0 or the GNU\nLesser General Public License (LGPL) Version 2.1. A copy of each\nsuch license is available at\nhttp:\/\/www.opensource.org\/licenses\/gpl-2.0.php and\nhttp:\/\/www.opensource.org\/licenses\/lgpl-2.1.php\n<\/pre>\n
    At this time we had to configure the IP of the FI:<\/p>\n
    switch(boot)# conf terminal \n\nEnter configuration commands, one per line.  End with CNTL\/Z.\n\nswitch(boot)(config)# interface mgmt 0\n\n\nswitch(boot)(config-if)# ip address 10.0.0.2 255.255.255.0 \n\n\nswitch(boot)(config-if)# no shutdown \n\n\nswitch(boot)(config-if)# exit\n\n\nswitch(boot)(config)# ip default-gateway 10.0.0.254\n\n\nswitch(boot)(config)# exit<\/pre>\n
    To get the Debug Plugin on the switch, you have to get it from a TFTP Server<\/p>\n
    switch(boot)# copy tftp:\/\/[my-tftp-ip]\/ucs-dplug.5.2.3.N2.2.27c.gbin workspace:debuug_plugin\/ucs-dplug.5.2.3.N2.2.27c.gbin\n\nTrying to connect to tftp server......\nConnection to server Established. Copying Started.....\n|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\|\/-\\\nTFTP get operation was successful\nCopy complete, now saving to disk (please wait)...\n\nswitch(boot)#<\/pre>\n
    In the following step we unmounted the Filesystems and repaired them:<\/p>\n
    witch(boot)# copy workspace:debug_plugin\/ucs-dplug.5.2.3.N2.2.27c.gbin xy\n\nCopy complete, now saving to disk (please wait)...\n\nswitch(boot)# load xy\n\nLoading plugin version 5.2(3)N2(2.27c)\n###############################################################\n  Warning: debug-plugin is for engineering internal use only!\n  For security reason, plugin image has been deleted.\n###############################################################\nSuccessfully loaded debug-plugin!!!\nLinux(debug)# umount \/dev\/mtdblock2\nLinux(debug)# umount \/dev\/mtdblock3\nLinux(debug)# umount \/dev\/sda3\nLinux(debug)# umount \/dev\/sda4\nLinux(debug)# umount \/dev\/sda5\nLinux(debug)# umount \/dev\/sda6\nLinux(debug)# umount \/dev\/sda7\numount: \/dev\/sda7: not mounted\nLinux(debug)# umount \/dev\/sda8\nLinux(debug)# umount \/dev\/sda9\numount: \/dev\/sda9: not found\nLinux(debug)# e2fsck -y \/dev\/sda3\ne2fsck 1.35 (28-Feb-2004)\n\/dev\/sda3: clean, 1746\/2125760 files, 1832918\/4247184 blocks\nLinux(debug)# e2fsck -y \/dev\/sda7\ne2fsck 1.35 (28-Feb-2004)\nCouldn't find ext2 superblock, trying backup blocks...\n\/dev\/sda7 was not cleanly unmounted, check forced.\nPass 1: Checking inodes, blocks, and sizes\nPass 2: Checking directory structure\nPass 3: Checking directory connectivity\nPass 4: Checking reference counts\nPass 5: Checking group summary information\nFree blocks count wrong for group #0 (24237, counted=21663).\nFix? yes\n\nFree blocks count wrong for group #1 (32461, counted=32442).\nFix? yes\n\nFree blocks count wrong for group #3 (32461, counted=32459).\nFix? yes\n\nFree blocks count wrong for group #8 (30266, counted=24180).\nFix? yes\n\nFree blocks count wrong for group #13 (32463, counted=32462).\nFix? yes\n\nFree blocks count wrong for group #14 (32462, counted=32463).\nFix? yes\n\nFree blocks count wrong for group #23 (32463, counted=32460).\nFix? yes\n\nFree blocks count wrong for group #26 (32460, counted=32456).\nFix? yes\n\nFree blocks count wrong for group #27 (32461, counted=32460).\nFix? yes\n\nFree blocks count wrong for group #29 (32463, counted=32462).\nFix? yes\n\nFree blocks count wrong (982140, counted=973450).\nFix? yes\n\nFree inodes count wrong for group #0 (9684, counted=9682).\nFix? yes\n\nFree inodes count wrong for group #1 (9696, counted=9693).\nFix? yes\n\nDirectories count wrong for group #1 (0, counted=1).\nFix? yes\n\nFree inodes count wrong for group #3 (9696, counted=9694).\nFix? yes\n\nDirectories count wrong for group #3 (0, counted=2).\nFix? yes\n\nFree inodes count wrong for group #8 (9689, counted=9687).\nFix? yes\n\nFree inodes count wrong for group #13 (9696, counted=9695).\nFix? yes\n\nDirectories count wrong for group #13 (0, counted=1).\nFix? yes\n\nFree inodes count wrong for group #14 (9695, counted=9696).\nFix? yes\n\nDirectories count wrong for group #14 (1, counted=0).\nFix? yes\n\nFree inodes count wrong for group #23 (9696, counted=9692).\nFix? yes\n\nDirectories count wrong for group #23 (0, counted=1).\nFix? yes\n\nFree inodes count wrong for group #26 (9693, counted=9689).\nFix? yes\n\nDirectories count wrong for group #26 (1, counted=2).\nFix? yes\n\nFree inodes count wrong for group #27 (9696, counted=9695).\nFix? yes\n\nDirectories count wrong for group #27 (0, counted=1).\nFix? yes\n\nFree inodes count wrong for group #29 (9696, counted=9694).\nFix? yes\n\nDirectories count wrong for group #29 (0, counted=1).\nFix? yes\n\nFree inodes count wrong (300543, counted=300523).\nFix? yes\n\n\n\/dev\/sda7: ***** FILE SYSTEM WAS MODIFIED *****\n\/dev\/sda7: 53\/300576 files (1.9% non-contiguous), 28596\/1002046 blocks\nLinux(debug)# e2fsck -n -f \/dev\/sda8\ne2fsck 1.35 (28-Feb-2004)\nPass 1: Checking inodes, blocks, and sizes\nPass 2: Checking directory structure\nPass 3: Checking directory connectivity\nPass 4: Checking reference counts\nPass 5: Checking group summary information\n\/dev\/sda8: 19\/501952 files (0.0% non-contiguous), 25374\/1002046 blocks\nLinux(debug)# \nLinux(debug)# e2fsck -n -f \/dev\/sda9\ne2fsck 1.35 (28-Feb-2004)\ne2fsck: No such file or directory while trying to open \/dev\/sda9\n\n\nThe superblock could not be read or does not describe a correct ext2\nfilesystem.  If the device is valid and it really contains an ext2\nfilesystem (and not swap or ufs or something else), then the superblock\nis corrupt, and you might try running e2fsck with an alternate superblock:\n    e2fsck -b 8193 <device>\n\nLinux(debug)# \nLinux(debug)# \nLinux(debug)# tune2fs -j \/dev\/sda3\ntune2fs 1.35 (28-Feb-2004)\nThe filesystem already has a journal.\nLinux(debug)# reboot\n\nINIT: \nINIT: Sending processes the TERM signal\n\nLinux(debug)# \nINIT: \/isanboot\/sbin\/loadplugin: line 172:  1490 Hangup                  $AUTORUN\n\nswitch(boot)# Sending all processes the TERM signal... \n\n\nSending all processes the KILL signal... \nSaving random seed:  \nSyncing hardware clock to system time \nUnmounting file systems:  \nmount: you must specify the filesystem type\nmount: \/var not mounted already, or bad option\nPlease stand by while rebooting the system...\n[ 1131.078240] Restarting system.\n[ 1131.114620] machine restart\n[ 1131.147864] Resetting board (uc)<\/pre>\n
    After two reboots the FI has it’s config<\/p>\n
    UCS1-B# show cluster state \nCluster Id: 0x357b36b2b45611e1-0xbaac547fee935324\n\nB: UP, SUBORDINATE\nA: UP, PRIMARY\n\nHA NOT READY\nWaiting for response from device.\nDevice count, expected: 3, active: 2\nDetailed state of the device selected for HA storage:\nChassis 1, serial: MYSERIALNO, state: inactive\nChassis 2, serial: MYSERIALNO, state: active\nChassis 3, serial: MYSERIALNO, state: active\n\nFabric B, Unable to connect to local chassis-shared-storage management interface\n:\nMYSERIALNO\n\nWarning: there are pending management I\/O errors on one or more devices, failove\nr may not complete\nUCS1-B# 2017 Oct 31 15:36:27 UCS1-B %$ VDC-1 %$ %SATCTRL-FEX1  -2-SATCTRL: IOM-1   Module 1: Cold boot\n\n2017 Oct 31 15:36:34 UCS1-B %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 1 is online (Serial number )\n\n2017 Oct 31 15:36:34 UCS1-B %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-1 On-line\n\n2017 Oct 31 15:36:34 UCS1-B %$ VDC-1 %$ %CALLHOME-2-EVENT: FEX_ONLINE\n\n2017 Oct 31 15:36:34 UCS1-B %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 1 is online (Serial number )\n\n2017 Oct 31 15:40:22 UCS1-B %$ VDC-1 %$ %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 3370%$ Interface vfc1217 is down (Error disabled)  server 1\/5, VHBA vHBA_0B \n\n2017 Oct 31 15:40:22 UCS1-B %$ VDC-1 %$ %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 3370%$ Interface vfc1295 is down (Error disabled)  server 1\/7, VHBA vHBA_0B \n\n2017 Oct 31 15:40:22 UCS1-B %$ VDC-1 %$ %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 3370%$ Interface vfc1177 is down (Error disabled)  server 1\/6, VHBA vHBA_0B \n\n2017 Oct 31 15:40:23 UCS1-B %$ VDC-1 %$ %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 3370%$ Interface vfc1153 is down (Error disabled)  server 1\/8, VHBA vHBA_0B \n\n2017 Oct 31 15:40:24 UCS1-B %$ VDC-1 %$ %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 3370%$ Interface vfc1569 is down (Error disabled)  server 1\/1, VHBA vHBA_0B \n\n\nUCS1-B#<\/pre>\n
    At this point the TAC Engineer told me he fixed this case.<\/p>\n
    Whooot???? Does not look like everything is fine! Ok, the FI has his config, but there are still errors.<\/p>\n
    Nerverless I had to open a new TAC Case to get this errors fixed:<\/p>\n
    [FSM:STAGE:RETRY:]: external VM manager extension-key configuration on local fabric(FSM-STAGE:sam:dme:ExtvmmMasterExtKeyConfig:SetPeer) F16898<\/strong><\/p>\n
    [FSM:STAGE:REMOTE-ERROR]: Result: service-unavailable Code: unspecified Message: Error syncing extension key(sam:dme:ExtvmmMasterExtKeyConfig:SetPeer) F78338<\/strong><\/p>\n
    [FSM:STAGE:REMOTE-ERROR]: Result: service-unavailable Code: unspecified Message: Error syncing extension key(sam:dme:ExtvmmProviderConfig:SetPeer) F78319<\/strong><\/p>\n
    [FSM:STAGE:RETRY:]: external VM manager configuration on peer fabric(FSM-STAGE:sam:dme:ExtvmmProviderConfig:SetPeer) F16879<\/strong><\/p>\n
    [FSM:FAILED]: external VM manager extension-key configuration(FSM:sam:dme:ExtvmmMasterExtKeyConfig). Remote-Invocation-Error: Error syncing extension key F999938<\/strong><\/p>\n
    [FSM:FAILED]: external VM manager configuration(FSM:sam:dme:ExtvmmProviderConfig). Remote-Invocation-Error: Error syncing extension key F999919<\/strong><\/p>\n
    At this time I was very disappointed with the TAC Support.<\/p>\n
     <\/p>\n
    To got further I found the Bug CSCvf27661<\/a><\/p>\n
    So I opened a new TAC Case to exchange those SSH-Keys.<\/p>\n
    First to see the error:<\/p>\n
    UCS1-A# scope system\nUCS1-A \/system # show managed-entity detail\n\nManaged Entity:\n    Fabric ID: A\n    Leadership: Primary\n    State: Up\n    Umbilical State: Full\n    HA Ready: Yes\n    SSH Internal Root Pub Key Checksum: aLongChecksum1\n    SSH Internal Root Pub Key Size: 225\n    SSH Internal Auth Keys Checksum: aLongChecksum2\n    SSH Internal Auth Keys Size: 225\n    SSH Internal Keys Status: Matched\n\n    Fabric ID: B\n    Leadership: Subordinate\n    State: Up\n    Umbilical State: Full\n    HA Ready: Yes\n    SSH Internal Root Pub Key Checksum: aLongChecksum3\n    SSH Internal Root Pub Key Size: 219\n    SSH Internal Auth Keys Checksum: aLongChecksum1\n    SSH Internal Auth Keys Size: 225\n    SSH Internal Keys Status: Mismatched<\/pre>\n
    Get the keys machting on both<\/strong> systems and edit it with vi \ud83d\ude09<\/p>\n
    Linux(debug)# cat \/root\/.ssh\/id_rsa.pub\nssh-rsa AveryLongSSHKey1== root@(none)\nLinux(debug)# cat \/var\/home\/samdme\/.ssh\/authorized_keys\nssh-rsa AveryLongSSHKey2== root@(none)\n<\/pre>\n
    I cant say how wired the TAC Engineer does this, it took more than one hour to edit this two keys. But he also did a mistake! Be sure your key ends with “root@(none)”.<\/p>\n
    In my case the “(none)” was missing in one key. I changed it after the TAC Session and get all errors gone.<\/p>\n
     <\/p>\n
    Hope this helps you …<\/p>\n
    Leave a comment and share!<\/p>\n
     <\/p>\n
    UPDATE:<\/span><\/p>\n
    Please also check CSCva31113<\/a>. This explained my issue exactly.<\/p>\n","protected":false},"excerpt":{"rendered":"
    The last days I tried to upgrade my UCS Domain from 2.2.7(c) to 2.2.8(g). I have two UCS Domains. One of them went through the upgrade fine, the other one not :\/ See what happend and how we fixed it …<\/p>\n","protected":false},"author":1,"featured_media":242,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[3],"tags":[],"class_list":["post-862","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cisco-ucs"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/posts\/862","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=862"}],"version-history":[{"count":24,"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/posts\/862\/revisions"}],"predecessor-version":[{"id":1107,"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/posts\/862\/revisions\/1107"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=\/wp\/v2\/media\/242"}],"wp:attachment":[{"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=862"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=862"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vblog.hochsticher.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=862"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}