Hyper-V: Cluster Shared Volumes, SQL Server, & FlashArray Snapshots

Many (arguably most) are currently evaluating hypervisor alternatives to VMware, and one obvious contender is obviously Microsoft Hyper-V. And if you’re a Pure Storage customer today, one thing you may be wondering, is whether you can keep utilizing storage-array snapshots to do things like replicate SQL Server data from one VM to another. The goods news is yes, absolutely! Let’s take a look at a common scenario as an example.

TL;DR – Show Me the Code!

A PowerShell example to orchestrate the above is now available up on Pure’s Github in the SQL Server Scripts repository. In addition to having the proper setup detailed below, read the README file’s Important Usage Notes section which will outline other pre-requisites.

If you just want to take Hyper-V CSV snapshots, without SQL Server involved, check out this Github repository, where I’ve shared a PowerShell solution. This is particularly useful if you want to clone an entire VM and present a second iteration of it back to a Hyper-V cluster.

And to see this in action, watch this narrated demo recording!

Updating Non-Prod Databases with a New Prod Copy

One extremely common task and time sink for DBAs is needing to take a backup of a production database and restore it to a non-production environment. Because it is a size of data operation, it can be quite time consuming. To help with that, Pure offers solutions that I’ve showcased, that leverage FlashArray snapshots, to orchestrate such workflows nearly instantaneously. But because (anecdotally) ~90% of all SQL Servers are on VMware, our examples were written for VMware. So now I want to show you how to do it in Hyper-V.

Hyper-V: Key Tidbits Overview

In Hyper-V, virtual disks take the form of VHDX files that reside on a Cluster Shared Volume (CSV) (or generally pre-Windows Server 2012, VHD’s that reside on a Clustered Disk). While CSVs were around in Windows Server 2008R2, they only really became useful and relevant in their second iteration, released with Windows Server 2012 and this linked blog has a fantastic deep dive into the CSV stack.

If you’re familiar with VMware, then I would say that a CSV is somewhat analogous to a VMFS datastore. A VMFS datastore contains one or more VMDK files, with each file representing a virtual disk for a virtual machine. With Hyper-V, each VHDX file is like a VMDK file.

So when you take a snapshot on FlashArray, you’re taking a snapshot of the entire volume – the CSV file in Hyper-V or the VMFS datastore in VMware (unless you’re using vVols – a separate conversation for another time).

The Setup & Workflow

For this, imagine that you have two SQL Servers: SQL-Prod and SQL-NonProd. These are Hyper-V VMs and may reside on the same CSV or different CSVs, it does not really matter. The idea is that we do not want to snapshot the ENTIRE VM, but instead isolate 1 or more SQL Server user databases, to clone from Prod to non-Prod.

So what you’ll do is create two more CSV’s, one for each SQL Server VM. We’ll call them CSV-UserDB-Prod and CSV-UserDB-NonProd for sake of discussion. Inside each CSV, will be the VHDX file(s) that correspond to the virtual disks that contain your user database data files and log files. If you have say, a D:\ drive for data and L:\ drive for log, then you should expect to have two VHDX files in your CSV.

The workflow is that we will then be taking a snapshot of just CSV-UserDB-Prod, and cloning that snapshot and overlaying CSV-UserDB-NonProd, thus updating the underlying data and files. Production will not be interrupted by at all by the snapshot, but because these are volume level operations, non-Prod will be disrupted.

Thanks for reading – happy snapshotting!

Hyper-V Cluster Shared Volumes + Write Caching

This blog post will probably never apply to anyone except me, but I’m writing it anyway.

Let’s say you have some VMware VMs… and decide to run Hyper-V hosts/cluster within those VMware VMs. And let’s say you start running into some headaches, taking crash-consistent snapshots and those snapshots not coming back in a consistent state as you would expect. That’s been my headache recently.

Write Caching?

So there’s several different layers of caching that can come into play, as documented here: “Hyper-V storage: Caching layers and implications for data consistency.” So I searched high and low, to try and figure out if I had unexpected write caching set up on my Cluster Shared Volumes. Looking at Disk Management -> Properties (of a CSV), one would normally see this:

Write-caching policy -> Enable write caching on the device

However… that is what one would see IF their respective Hyper-V host was on a bare-metal server. That’s NOT what one sees though, if the Hyper-V host resides on a VMware VM backed by a vVol (Virtual Volume). THIS is what you get…

Note that there are TWO policies now – Removal Policy

Removal Policy?! Where did that come from? Doing some digging, that is normally only present for removable storage like USB drives. And look at the default setting’s description… “enables write caching in Windows…”

So I disabled that on each of my Cluster Shared Volumes, bounced both Hyper-V hosts (not 100% sure if it was required but did so just to be sure), and HURRAY – my crash consistent snapshot code worked as expected!!!

One reason why this was so frustrating is that I was not doing most of this investigative work via the GUI, but rather via PowerShell commands. So I was digging around things like Cluster Parameters and turning CsvEnableBlockCache and CsvEnforceWriteThrough on and off (https://learn.microsoft.com/en-us/windows-server/failover-clustering/failover-cluster-csvs & https://learn.microsoft.com/en-us/previous-versions/windows/desktop/mscs/physical-disk-csvenforcewritethrough). But nuking this Removal Policy wound up doing the trick for me.

TL;DR

If you’re trying to disable any and all write caching, for whatever reason, be sure to check if Windows thinks the disk is removable media and has a Removal Policy applied (that also enables write caching).

Thanks for reading.