Veeam Backup & Replication VSS Errors

A few days ago, one of our VMs running on Hyper-V 2012 R2 became stuck and locked in a “Backup up…” status. We use Veeam Backup & Replication 7.0 and had noticed that this particular VM had been reverting to crash-consistent backups for the prior three days. The summary said it was a transient VSS error, so we didn’t dive deeper until it persisted. That’s when we saw it was stuck.

The problem with “stuck” is that Hyper-V won’t let it go. Users on Social TechNet discuss this issue here, but the nutshell is that it requires a reboot (often hard) of the Hyper-V host, because the VM process locks and the host won’t transition it, even fully shutdown at the guest level. Thus, we evacuated everything else and then power cycled the host. Windows Server 2012 R2 didn’t react so well to that, and subsequently required booting into Safe Mode to finally realize it was okay and able to boot normally.

veeam_vss_failedThat next backup window had a bunch of warnings about change block tracking (CBT) files (.avhdx) not matching, but it performed full backups fine. Not so the day after. Failed. Failed. Failed…

I maintenance mode’d and rebooted the Hyper-V hosts, restarted their VSS services, etc, but still they failed.

Then I tried a backup with our DPM server, which used to backup our Hyper-V VMs. It succeeded. So it wasn’t a host issue like I originally thought.

Maybe Veeam needed a reboot, since the VSS provider was working for DPM, so I rebooted the Veeam servers. Failed again.

This is the warning Veeam had been throwing when it failed the snapshots:

7/10/2014 7:04:07 PM :: Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing with failover). Details: Provider [{b5946137-7b9f-4925-af80-51abd60b20d5}] does not support shadow copies of volume [\\?\Volume{f8171d69-156f-455b-baba-d0c36c75ace3}\].
–tr:Failed to add volumes to the snapshot set.
–tr:Failed to perform pre-backup tasks.

The key portion is the provider listed: Microsoft Software Shadow Copy provider 1.0. Until two days ago, Veeam was using “Microsoft CSV Shadow Copy Provider”, but for some reason switched providers the day after the CBT warnings happened (after the Hyper-V host hard reboot). I didn’t take any note because that provider looked legitimate and was in the ‘vssadmin list providers’ list.

After much searching, a Veeam help page about configuring connected volumes came up on Google. I hadn’t been there before, but it looked promising. Maybe I could choose a different provider.

veeam_vss_null

Hmm… Null surely isn’t a valid provider selection. And clicking the drop down didn’t show the ‘1.0’ provider it had been trying to use. Whatever the cause, Veeam was confused.

veeam_vss

So on all CSV connected volumes on each/all hosts, I set things back to the way they were originally (when it worked). Finally, we had a solution. Retrying the job worked. Snapshots took, data flowed, all is well.

5 Comments

  1. Sebastien bouchard said:

    Thanks a lot !!! i have a brand new setup and i was having the same error. the VSS Provider was blank. after setting it to Microsoft CSV, it’s start working. i still have other unrelated problems but that, is another story !
    Thanks again

    March 19, 2015
    Reply
    • Chris said:

      Glad it helped!

      March 19, 2015
      Reply
  2. danang said:

    Hello Chris,
    i followed your instruction, but it not solved my problems
    i still have error with vss when backup a vm on hyper-v

    this is my log file :
    Physical host volume was not found for path ‘\\CSIJKTFS01\Hyper-V\vsvr\VSERVER\Virtual Hard Disks\VSERVER.vhdx’ on host ‘CSIJKTFS01’.
    [29.06.2015 09:50:42] Warning [RTS] [VSERVER] Skipping post-snapshot ctp flush for VM ‘VSERVER’, id ’50a134d8-9dd0-40f1-9d18-36d54f88a293′ : pre-snapshot ctp flush has failed.
    [29.06.2015 09:50:42] Warning [VSERVER] Skipping post-snapshot ctp flush for VM ‘VSERVER’, id ’50a134d8-9dd0-40f1-9d18-36d54f88a293′ : pre-snapshot ctp flush has failed.
    [29.06.2015 09:50:42] Warning [RTS] [VSERVER] Unable to create snapshot (Fileshare Provider) (mode: Crash consistent).
    [29.06.2015 09:50:42] Warning [VSERVER] Unable to create snapshot (Fileshare Provider) (mode: Crash consistent).
    [29.06.2015 09:50:42] Error Provider [{89300202-3cec-4981-9171-19f59559e0f2}] does not support shadow copies of volume [\\CSIJKTFS01\HYPER-V\].
    [29.06.2015 09:50:42] Error –tr:Failed to add volumes to the snapshot set.
    [29.06.2015 09:50:42] Error –tr:Failed to perform pre-backup tasks. (System.Exception)
    [29.06.2015 09:50:42] Error Provider [{89300202-3cec-4981-9171-19f59559e0f2}] does not support shadow copies of volume [\\CSIJKTFS01\HYPER-V\].
    [29.06.2015 09:50:42] Error –tr:Failed to add volumes to the snapshot set.
    [29.06.2015 09:50:42] Error –tr:Failed to perform pre-backup tasks. (System.Exception)
    [29.06.2015 09:50:42] Error [RTS] On-host snapshot was not created. Result: ERR: [Provider [{89300202-3cec-4981-9171-19f59559e0f2}] does not support shadow copies of volume [\\CSIJKTFS01\HYPER-V\].
    [29.06.2015 09:50:42] Error –tr:Failed to add volumes to the snapshot set.
    [29.06.2015 09:50:42] Error –tr:Failed to perform pre-backup tasks.]
    [29.06.2015 09:50:42] Error [VSERVER] On-host snapshot was not created. Result: ERR: [Provider [{89300202-3cec-4981-9171-19f59559e0f2}] does not support shadow copies of volume [\\CSIJKTFS01\HYPER-V\].
    [29.06.2015 09:50:42] Error –tr:Failed to add volumes to the snapshot set.
    [29.06.2015 09:50:42] Error –tr:Failed to perform pre-backup tasks.]
    [29.06.2015 09:50:42] Info [ScriptInvoker] Script enabled = ‘False’
    [29.06.2015 09:50:42] Info [ScriptInvoker] Creating empty invoker
    [29.06.2015 09:50:42] Info [CHvSnapshotHolder] Non shared snapshot created
    [29.06.2015 09:50:42] Info [RTS] [VSERVER] De-installing guest controls.
    [29.06.2015 09:50:42] Info [VSERVER] De-installing guest controls.
    [29.06.2015 09:50:42] Warning StopAliveThread: atop event was already destroyed or not created
    [29.06.2015 09:50:42] Info [VssGAConn] Uninstalling agent
    [29.06.2015 09:50:42] Info Enter CVeeamVssGAInstaller::Uninstall
    [29.06.2015 09:50:42] Info Impersonating network logon session.
    [29.06.2015 09:50:42] Info Impersonating network logon session.. Ok.
    [29.06.2015 09:50:42] Info Stopping service: [VeeamVssSupport].
    [29.06.2015 09:50:45] Info Unregistering service: [VeeamVssSupport].
    [29.06.2015 09:50:45] Info Deleting target folder: [\\172.25.1.83\ADMIN$\VeeamVssSupport]
    [29.06.2015 09:50:45] Info Leave CVeeamVssGAInstaller::Uninstall

    apreciate with your suggestion..
    thanks

    June 28, 2015
    Reply
    • Chris said:

      That looks like a different issue than I was documenting here, though they could be related. I see it using “Fileshare Provider” which is different than the Microsoft CSV provider. This post pertains to Hyper-V clusters with VMs on CSVs.

      If this context does apply, then it seems you need to change the VSS provider from “Fileshare” to “CSV”. Beyond that, I can’t really speak to your case more specifically without seeing it, etc. Sounds like a job for Veeam Support. Their forums may help, too.

      June 29, 2015
      Reply
  3. Nimda said:

    Hi, thanks for the tip.

    Rescan option in the Manage Volumes dialog fixed it automatically in my case.

    July 4, 2015
    Reply

Leave a Reply