Tag: Hyper-V

We received an update from our Dell team today. It looks like politics of who’s the root cause are going to make all of us suffer for another 6 months at least. Read on…

3Sept2014 BG – We investigated this issue. Looks like it is due to anomalous behavior of HyperV NDIS stack. Driver allocates MSIX interrupts for VMQs at load time and sets interrupt affinity bases on the RSS processor set returned by NdisGetRssProcessorInformation call as per MSDN documentation. None of the host CPU is in the list of RSS processors below the base RSS Processor number RssBaseProcNumber.

Later on NDIS specifies cpu0 when it send OID to allocate VMQ. Driver doesn’t find any MSIX interrupt to satisfy the VMQ allocate OID and hence driver fails the OID.

Microsoft Networking Technology Virtualization

We’ve confirmed with Dell Support that QLogic has identified a bug with the QLE82xx network driver (at least through version 5.3.12.0925) and Virtual Machine Queues (VMQ). As of August 15, 2014, QLogic reports that they have reproduced the issue, but have not resolved it.

We have a case open with Microsoft Hyper-V Support and that case number has been shared with Dell and QLogic to coordinate troubleshooting and support as the issue becomes more visible in the community. I’ll post updates as we have them. There is word of a beta driver at some point, which we’ve expressed interest in testing.

Networking Technology Virtualization

In Part 1, I laid out a brief summary of VMQ and an example of the configuration that is appropriate for our four-socket, ten-core Hyper-V host. Here in Part 2, I’ll unpack the issue we’re facing in spite of our textbook configuration.

Following the guidance in VMQ Deep Dive, Part 2, using the commands in Part 1 of this blog, we find the below queue list. The three line items are default queues given to the Hyper-V host (HV01) on its physical ports and the related logical switch. We should be seeing the VMs on this host in that list as well, but we aren’t.

vmq_ps_not_zero

The Windows System event log shows the following error, which seems to be the issue with queuing.

vmq_oid_failed

Microsoft Networking Technology Virtualization

Microsoft has been gaining ground in the virtualization sphere one step at a time since Hyper-V first premiered. While the some increments were negligible (or merely painstakingly obvious), they achieved significant breakthroughs in late 2013 with the release of all things “2012 R2”. The puzzle piece on which we’ll focus here is VMQ (specifically dynamic VMQ, or dVMQ).

get-netadaptervmqVMQ gives Hyper-V and System Center Virtual Machine Manager (VMM) Logical Switches what Receive Side Scaling (RSS) provides to physical servers; namely, it leverages multiple compute cores/interrupts to increase network traffic efficiency. The network teaming (or Load-Balancing Fail-Over, LBFO) configuration is important here, because it affects how VMQ maps queues to processors. The full table of possibilities is given halfway down the page of TechNet’s VMQ Deep Dive, Part 2. In a nutshell, some configurations need NIC queues to overlap the same processors (so that all queues are everywhere), while others need segregation (so every queue has its own unique core).

Microsoft Networking Technology Virtualization