VMware ESXi Round-Robin NMP IOPS=1

When we started our initial foray into the all-flash array space, we had to put on the brakes when the “best practice” recommendations started flying from the SEs and guides. In a perfect world, we’d be entirely on the new array (Pure Storage was first), but migration is a necessary process. We also wanted a clear back to go back if POCs failed. The recommendation for IOPS before changing paths with Round-Robin native multipathing (NMP) was one of those settings.

From the EMC XtremIO Storage Array User Guide 2.4:

For best performance, it is recommended to do the following:

  • Set the native round robin path selection policy on XtremIO volumes presented to the ESX host.
  • Set the vSphere NMP Round Robin path switching frequency to XtremIO volumes from the default value (1000 I/O packets) to 1.

These settings ensure optimal distribution and availability of load between I/O paths to the XtremIO storage.

I never pursued that path to see if HP 3PAR would tolerate it, since other settings were clearly incompatible, but apparently HP came to their own realization on the matter. That said, please take caution with environments running more than just these two arrays, and watch out for the other “best practices” for all-flash arrays. Setting the queue depth to max (256) or raising concurrent operations to 64 will likely overwhelm or cause I/O loss when non-flash arrays are under pressure.

Disclaimer aside, environments with both/only HP 3PAR and EMC XtremIO storage arrays connected to the same vSphere hosts, the need for compromise is no longer necessary for IOPS. Round Robin NMP can now use a value of 1 for both arrays.

The setting of IOPs=1 is a new recommended initial value as documented in the HP technical white paper HP 3PAR StoreServ Storage and VMware vSphere 5 Best Practice (HP document #4AA4–3286ENW). You can change this setting to suit the demands of various workloads. ~HP 3PAR VMware ESX Implementation Guide, Updated July 17, 2014

Having said all this, the original conflict of interests only applies when using Persona 6 (Generic) on HP 3PAR arrays, since that persona uses SATP rule, “VMW_SATP_DEFAULT_AA” (the same as XtremIO). If you update 3PAR to use Persona 11 (ESX 4.x, 5.x), the SATP rule changes to VMW_SATP_ALUA. The different rules use different default multipathing policies and IOPS settings. Thus, they are not in conflict if one is changed.

To check the SATP rule in use, you can use the vSphere Client: Configuration tab > Storage. Select a datastore, click “Properties” in the bottom half, click “Manage Paths”, and look at the “Storage Array Type” in the top “Policy” box.


In the CLI:

# esxcli storage nmp device list


That Round-Robin setting with the ALUA rule above wasn’t default and I wasn’t aware of how to change the default until tonight. That meant that I’ve done a lot of clicking to change each LUN over from the ALUA default of Most Recently Used (MRU) to Round Robin (RR).

HP recommends a custom SATP rule as long as you’re above ESX 4.0 (4.0 doesn’t play well with custom rules, apparently). Here’s the provided command for ESXi 5.x. As you can see, it only applies to devices with “3PARdata” in the name.

# esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -P "VMW_PSP_RR" -O iops=1 -c "tpgs_on" -V "3PARdata" -M "VV" -e "HP 3PAR Custom iSCSI/FC/FCoE ALUA Rule"




As noted in the HP guide, a reboot is necessary:

The host must be rebooted for SATP rule creation, changes, additions, or removals to take effect on existing, previously presented devices.

Here’s the EMC-recommended command for XtremIO as well:

# esxcli storage nmp satp rule add -c tpgs_off -e "XtremIO Active/Active" -M XtremApp -P VMW_PSP_RR -O iops=1 -s VMW_SATP_DEFAULT_AA -t vendor -V XtremIO

Back to HP 3PAR, here’s the validation that the reboot was effective (note the “iops=1”):


If you don’t see any change or your rule list doesn’t show the new rule, verify (or retype) the command and the symbols therein. I’ve found that copying commands from docs/pages have a way of switching to non-ASCII characters. My first attempt with the custom rule didn’t take until I retyped the hyphens and quotes.

Hope you enjoy a bit of both worlds of best practices.

One Comment

  1. […] Here’s the short and sweet for configuring the best practice SATP rule for XtremIO storage on ESXi 5.5 using PowerCLI (5.8 Release 1, in my case). I can’t claim any credit beyond aggregation and adaptation: the parameters are from the XtremIO user guide and the script comes from VirtuallyHyper.com (thanks!). See my earlier post about the SATP rule itself and how to manually implement it: VMware ESXi Round Robin NMP IOPS=1. […]

    October 10, 2014

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.