A few days ago, one of our VMs running on Hyper-V 2012 R2 became stuck and locked in a “Backup up…” status. We use Veeam Backup & Replication 7.0 and had noticed that this particular VM had been reverting to crash-consistent backups for the prior three days. The summary said it was a transient VSS error, so we didn’t dive deeper until it persisted. That’s when we saw it was stuck.
The problem with “stuck” is that Hyper-V won’t let it go. Users on Social TechNet discuss this issue here, but the nutshell is that it requires a reboot (often hard) of the Hyper-V host, because the VM process locks and the host won’t transition it, even fully shutdown at the guest level. Thus, we evacuated everything else and then power cycled the host. Windows Server 2012 R2 didn’t react so well to that, and subsequently required booting into Safe Mode to finally realize it was okay and able to boot normally.
That next backup window had a bunch of warnings about change block tracking (CBT) files (.avhdx) not matching, but it performed full backups fine. Not so the day after. Failed. Failed. Failed…
I maintenance mode’d and rebooted the Hyper-V hosts, restarted their VSS services, etc, but still they failed.
Then I tried a backup with our DPM server, which used to backup our Hyper-V VMs. It succeeded. So it wasn’t a host issue like I originally thought.
