How to Disable/Enable Changed Block Tracking (CBT) on Running VMs

If you use a backup product that leverages VMware’s changed block tracking (CBT), you have probably also found cases when CBT wasn’t the right fit. In the world of EMC Avamar, I’ve found that VM image-level backups will slow to a crawl if enough blocks change but don’t quite reach the level (25%) necessary to automatically revert to a full backup.

When I created a case with EMC Support, they dug into the logs and then pointed to best practices that recommend disabling CBT when more than 10,000 blocks regularly change between backups. The problem I hit next was that the top result and KB for enabling/disabling CBT was a VMware post stating that step 1 was to power off the VM. Backups are running long and the next maintenance window isn’t for two weeks. Hmm…

  1. Avamar method
  2. PowerCLI method

1. Avamar. Then I recalled that Avamar seems to have no issue turning on CBT hot when adding VMs to its inventory. If it can turn it on, how about turning it off? From the GUI, the only time it exposes that option is adding a “New Client” in the Administration panel. To be new, it can’t exist, which either means retiring the active client or deleting it. Deleting it worked smoothly, but also wiped out all backups (not a good idea); retiring seemed to hang when adding it back, but sometimes worked. Getting closer…

EMC suggested trying the proxycp.jar utility on the utility node. This tool does a lot of fun stuff with VM clients and their related backup proxies. To download the latest version (important), use FTP and the link below:

ftp://avamar_ftp:[email protected]/software/scripts/proxycp.jar

Then, upload it via SFTP/SCP to your Avamar utility node and ‘sudo cp’ it to /usr/local/avamar/bin. After that, SSH to the utility node and run the following commands to disable CBT and then verify that it worked (or run first to check status):

proxycp.jar --cbtstatus --disablecbt --vm
proxycp.jar –cbtstatus –disablecbt –vm <vm_name>
proxycp.jar --cbtstatus --vm
proxycp.jar –cbtstatus –vm <vm_name>

2. PowerCLI. “Benj” posted a quick script on this back in 2012 with the full post available here:

http://www.itwalkthru.com/2012/03/disabling-or-enabling-vmware-change.html

Others have since pointed to Veeam KB1940 for validation of this method, and curiously, the exact same script is given. Either both pulled from the same source (albeit Veeam was two years later), or Veeam pulled from Benj, too. Regardless, here’s the gem:

Get the VMs with CBT enabled:

$vms=get-vm | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}

Create a VM Specification to apply with the desired setting:

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec 
$spec.ChangeTrackingEnabled = $false

Apply the specification to each VM, then create and remove a snapshot:

foreach($vm in $vms){ 
$vm.ExtensionData.ReconfigVM($spec) 
$snap=$vm | New-Snapshot -Name 'Disable CBT' 
$snap | Remove-Snapshot -confirm:$false}

Check for success:

get-vm | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}

If others in the community have any warnings concerning these hot-mods, feel free to comment. Hopefully this can help those who have otherwise been limited to downtime changes.

2 Comments

  1. Sam Chow said:

    My question is about the snapshot process.
    I have a SQL sever on which I need to reset CBT. Will this command create a complete snapshot of the server ? My concern is that the deltas will not get consolidated and removed during business hours. Server is 4 disks of 2TB and 500GB each.

    March 8, 2017
    Reply
    • Chris said:

      Sam, yes it will create a complete/full snapshot and backup, but your concern is valid. A new full backup may incur significant consolidation time, depending on your backup software/target.

      The context of this article was Avamar, which wasn’t always quick about taking full backups. In 2015, I replaced Avamar with Rubrik (www.rubrik.com; for whom I now work), which addresses this pain point of customers (consolidation challenges), among other benefits.

      Let me know if you have any other questions!

      March 12, 2017
      Reply

Leave a Reply