How To Upgrade to ESXi 6.0 And Overcome “Incompatible” Status

With the release of ESXi 6.0 Update 1a, which fixed the network connectivity issue that plagued all ESXi 6.0 releases until October 6, I have begun my own journey from 5.5 to 6.0. I’m taking a new approach for me, though, as I use Update Manager to perform an upgrade rather than the fresh installs I have always preferred.

Why? Because I learned at VMworld 2015 from the authorities (designers) that upgrading is actually VMware’s recommended path. You can read more from my notes on session INF5123.

What follows below assumes that you have already rebuilt or upgraded to vCenter 6.0 Update 1. In Update 1, the Web Client now supports Update Manager so that everything can be performed there. No more thick client! Now if we can just get rid of Flash…

Step 1: Import ESXi Image

From the home landing page of the vSphere Web Client, navigate here:

  • Update Manager
    • Select an Update Manager server
      • Go to Manage
        • Then ESXi Images
          • Import ESXi Image…
            • Browse to the ISO

esxi6_import

Step 2: Create A Baseline

Navigate from the ESXi Images sub-tab over to Hosts Baselines and click the + to create a new baseline:

esxi6_baseline_1

esxi6_baseline_2

esxi6_baseline_3

Step 3: Attach The Baseline

Returning to the home landing page of the vSphere Web Client, navigate here:

  • Hosts and Clusters
    • <Your Cluster Name>
      • Manage
        • Update Manager
          • Attach Baseline…

esxi6_attach

Step 4: Scan for Updates

With the baseline attached and selected under Attached Baselines:, click
esxi6_scan_button and accept the defaults (Patches and Extensions is optional and irrelevant to the host upgrade). Scanning can take several minutes, so check back after a while.

esxi6_scan

Step 5: Fix Hosts with Incompatible Status (may not apply)

In my environment, I had two hosts in one cluster and one host in another cluster that scanned and showed to be “Incompatible” for the host upgrade baseline. That didn’t bode well at all until I checked the detailed status and found it to be an unused package conflict. If you find yourself in the same place, read on. If not, skip to Step 6.

Before continuing with any “fix” steps below, Enter Maintenance Mode on your host. Removing VIBs will require a reboot.

On the cluster’s Manage > Update Manager pane, the bottom half shows the statuses after a scan. Incompatible hosts are on the third tab and look like this:

esxi6_incompat_cluster

Click on the host to link to its Summary page. One of the widgets on the Summary is Update Manager Compliance.

esxi6_incompat_widget

Click the Detailed Status link to hop to the host’s Update Manager page. Then select the attached baseline to see the details.

esxi6_incompat_detail

Since my host doesn’t have any Infiniband devices, the Mellanox VIB is unnecessary and can be removed. To do that, we need to connect to the console. I prefer SSH, but feel free to use another method.

To start SSH (stopped by default), switch from the Update Manager sub-tab to Settings. In the vertical menu below, go to Security Profile and then collapse the Incoming Connections and Outgoing Connections to quickly “scroll up” the Services section. Edit and start SSH.

esxi6_ssh

Using your SSH client of choice (I use PuTTY), connect to the host as root or a similarly privileged account and run the following commands.

~ # esxcli software vib list |grep Mellanox

This should return:

net-mlx4-en 1.9.9.0-1OEM.550.0.0.1331820 Mellanox VMwareCertified 2014-08-06

Using the VIB name, remove it with the following command. The result should follow. Make sure “vibname” is prefixed by double dashes. Web formatting can sometimes obscure that, which will lead to ESXi rejecting the command (i.e. from a single dash or em-dash).

~ # esxcli software vib remove --vibname net-mlx4-en
Removal Result
 Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
 Reboot Required: true
 VIBs Installed:
 VIBs Removed: Mellanox_bootbank_net-mlx4-en_1.9.9.0-1OEM.550.0.0.1331820
 VIBs Skipped:

At this point, it’s time for a reboot. Double-check to ensure your host is in maintenance mode before running “reboot” from SSH or the Web Client.

After rebooting and connecting to the host via the Web Client, return to the Manage > Update Manager page and Scan for Updates… again. The compliance status should change from “Incompatible” to “Non-Compliant”. Once there, it’s time for the upgrade!

Step 6: Upgrade to ESXi 6.0 Update 1a

Now is a good time to (re)make sure that the image you uploaded is indeed the latest from October 2015 (or newer). Anything before that puts your host at risk of network or security issues.

Once that is confirmed, on the Update Manager page where you previously scanned, click the Host Upgrade baseline and then Remediate…

esxi6_remediate_1
Select the Upgrade baseline
Select the host(s) to upgrade
Select the host(s) to upgrade
Accept the EULA terms
Accept the EULA terms
Leave unchecked (you want to see warnings)
Leave unchecked (you want to see warnings)
Name the task and choose to "Run Now" unless it should be scheduled for later
Name the task and choose to “Run Now” unless it should be scheduled for later
Accept defaults for maintenance mode and leaving VMs powered on
Accept defaults for maintenance mode and leaving VMs powered on
Check to disable HA Admission Control (depending on thresholds, this can hold up the task)
Check to disable HA Admission Control (depending on thresholds, this can hold up the task)
Review the summary and kick it off!
Review the summary and kick it off!

Use your out-of-band management (iLO, DRAC, IPMI) to monitor the upgrade progress. In my case, the story didn’t wrap up neatly, and the DRAC revealed an endless loop due to driverless hardware.

Funny thing with management networks--they need NICs!
Funny thing with management networks–they need NICs!

It should come as no surprise that my QLogic 8262 converged network adapters (CNAs) had no drivers in the native ESXi 6.0 ISO. Since my management network resides in a port group on those adapters, the upgrade process errored out and rebooted repeatedly with the above message.

The solution was/is to boot from the ESXi 6.0 Update 1a ISO and perform/complete the upgrade that way. Just select the disk where ESXi 5.5 resides and choose Upgrade instead of Install. In less than a minute, the installer will complete and prompt you to reboot. That will return you to the ESXi console.

How can we get around this hiccup?

As I see it, there’s two options:

  1. Create a custom-build of ESXi 6.0 with the additional drivers
  2. Move the Management Network to natively-supported NICs

My environment is relatively small, so I’m going with option 0 (or 3, but neither 1 nor 2) and upgrading via ISO. I have some copper to satisfy option 2, but it isn’t VLAN’d the same way so it won’t be transparent. It’s easier to just hand-hold the install and get it done. Next major upgrade I’ll hopefully be off of these QLogic adapters and it will be a moot point.

Step 7: Scan for Updates (post-6.0)

Once the host is back online, you’ll want to make sure that the Critical Host Patches baseline is attached, and optionally the Non-Critical Host Patches baseline. Then Scan for Updates… again and apply the relevant patches.

After the required reboot, make sure networking looks good and storage is attached. Check your hardware for firmware and driver updates. Finally, take the host out of maintenance mode and move on to the next host.

Hopefully in your case you’ll be able to vet the process on one host and then automate the remaining hosts with Update Manager.

One Comment

  1. Josh Glassberg said:

    Thanks! I used this a few times. Upgrading from 5.5 Build 1892794 to 6.0 Build 3620759
    esxcli software vib remove –vibname net-mlx4-en Worked for me!

    March 23, 2016
    Reply

Leave a Reply