If you’re thinking about Windows Server 2012, or maybe you aren’t, but you have a lot of uncompressed archive data like log files, syslog, documents, etc, check out the new deduplication feature. It’s a real charmer.
As an example for this post, last Friday I undertook to apply this new functionality to our syslog server. It runs Kiwi Syslog (now owned by SolarWinds) and has held about 400GB of logging that we’ve pruned at the 3-month mark to keep it under control. With this in mind, I spun up a new VM, imaged it from our SCCM server with Windows Server 2012, and signed on.
Being that it was Friday and I wanted to accomplish this quickly without transferring 400GB+ between servers (from near-line drives), I decided to swing the VMDK disk over from the old server. So I exported the Kiwi config to .ini, stopped the Kiwi Syslog service, deactivated the license (so I could install it on the new server), installed Kiwi on the new server, and unmounted (marked “Offline”) the disk in Disk Management.
At that point, I edited the old VM (while running), removed the log disk (remove, NOT delete), edited the new VM, and added the existing VMDK from the old VM’s folder. On the new server in Disk Management, I made sure it was set to use the same drive letter as it was on the old server, and I imported the .ini file into Kiwi. Up to this point, the old server has held onto its IP address (to which all network devices send their logs).
It was time to go live, so I made sure Kiwi was running, changed the old server to DHCP, set the released IP on the new server, and watched the data start flowing in. But wait. We haven’t touched on dedupe.
In Server Manager, I clicked “Add roles and features” and Next’d my way to the Server Roles step. Expanding “File And Storage Services” and “File and iSCSI Services”, I found “Data Deduplication”. Check this box and away we go!
Now on the left side of Server Manager, I have “File and Storage Services”. Digging in there, I find “Disks” under “Volumes” on the left menu. So last Friday, I selected the data disk that I brought over from the old server (in the top pane), then right-clicked the “D:” volume in the lower “Volumes” pane. Finally I click “Configure Data Deduplication”, check the “Enable data deduplication” box, set my time frame (I used 7 days), and clicked OK.
I could have called it quits here, but I was eager to see the process start, so I fired up PowerShell and issued the “Start-DedupJob” command. From there, I watched the redundant bits disappear using the “Get-DedupStatus” command. Today, instead of 415GB of 500GB used, I have that amount free and have removed my syslog cleanup job. I doubt I’ll ever run out of disk space even after years of logs pile up.
Hope this story helps you. Have fun being efficient!