Under the supervision of VMware vCenter Server, ESX 4 thin provisioning is safe and reliable even for production workloads thanks to advanced storage accounting and built-in monitoring.  And even if the worst does happen — an unexpectedly full datastore — recovery is simple.

Now that you are convinced that VMware vSphere thin provisioning has a place in your data center, you may be wondering about performance tradeoffs.

It turns out that thin disks perform just about as well as thick disks.

New Performance Results

In this brand-new VMware vStorage Thin Provisioning performance study, VMware performance engineers compare thick and thin disk performance.  This must-read document covers several important topics, such as:

• I/O-intensive benchmarking with a 16-node ESX cluster
• File copy benchmarking with 2 ESX hosts
• Fragmentation impact
• Thin provisioning affect on co-located thick disks

Now, go forth and provision… thinly.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. Finding thin-provisioned virtual disks with PowerShell
3. Easy recovery from a full VMware ESX datastore
4. VMware Update Manager Performance and Practices

• Full accounting for all provisioned storage space and complete monitoring
• Automatic response to a nearly-full datastore through vCenter Alarms, PowerShell, and Storage VMotion

You may be nearly convinced to start using thin provisioning, but still wondering…

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically grow to accommodate additional storage demand.  When VMware ESX detects this condition, virtual machines in need of additional storage are instantly paused to prevent guest operating systems from failing. Conversely, VMs that that read and write to existing allocated storage blocks will continue running without issue — not all virtual machines will be paused just because a datastore is out of space.

If you ever find yourself in this situation, it’s not hard to fix.  Here is one simple approach, step-by-step:

1. Free up some space by deleting or moving files — ISO images or powered-off VMs would be perfect
2. Resume one of the paused VMs
3. Use Storage VMotion to move the disks for that VM to another datastore
4. Resume the remaining VMs

Watch the procedure in action:

Depending on the size and storage demand of each VM, additional migrations may be needed.  An alternative resolution would be to add additional space to the SAN LUN and grow the VMFS volume.

The Experiment

To simulate a sudden storage demand by the thin-provisioned VMs in the above video, I simply copied a large file from a network share to each Windows Server 2003 VM simultaneously.

For the curious, below is a PowerShell script for the task.  Run it from anywhere — it uses Sysinternals psexec to remotely initiate a file copy on each VM from a network share.

# VM naming convention combines this string with 2-digit number
$vmPrefix = "VM2003e-"
 
# UNC path to a large file that will be copied into each VM
$iso="\\fileserver\ISO\large.iso"
 
# credentials to download above file from \\fileserver
$user="domain\username"
$pass="pa55word"
 
# pass this function a list of numbers
function createVmList ($series) {
    $vmList = @()
 
    foreach ($id in $series) {
        $vmList +=  %{"$vmPrefix{0:00}" -f $id}
    }
    $vmList
}
 
# make sure Sysinternals psexec is in your path
function psexecOnVm ($cmd, $vmList) {
    $vmList | % {Invoke-Expression "psexec.exe \\$_ -d -u $user -p $pass $cmd"}
}
 
psexecOnVM "cmd /c copy /Y $iso c:\" (createVMlist (31..39))

VMware ESX is Resilient

You may have been surprised at how easy it is to recover from a full datastore — without so much as a guest OS reboot.  It’s a testament to the rock-solid architecture behind VMware ESX and VMFS.  No other virtualization platform comes close.  Try for yourself.  See what happens if a group of thin-provisioned Hyper-V virtual machines suddenly run out of storage — it’s not going to be pretty.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. PowerShell Prevents Datastore Emergencies
3. vSphere Thin-Provisioned Disk Performance
4. Finding thin-provisioned virtual disks with PowerShell

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. Register a LUN full of VMs with PowerCLI
3. Responsible Thin Provisioning in VMware vSphere
4. VMware vSphere 4 has a Snapshot Alarm

What’s new is not necessarily the technology — it’s the management.  In fact, veteran VMware ESX admins have been creating thin provisioned virtual disks for years — for controlled scenarios — by way of the vmkfstools command.

Before vSphere and ESX 4, however, thin disks came with risk — there was no simple way of accounting for the overcommitted storage on each LUN.  Even with multiple gigabytes of free space, a small gang of thin-provisioned virtual machines could quickly quickly grow to exceed datastore capacity during a sudden demand spike.

Complete Storage Accounting

Now in vSphere 4 there is a new element in the capacity section of the datastore summary tab that shows total provisioned space — the maximum potential growth of all virtual machines if thin provisioned disks were fully utilized:

Virtual machines with snapshots have the potential of consuming even more datastore space, so vSphere accounts for this condition, too.  Take a look at this VM Summary tab, where the total provisioned storage includes:

• Virtual hard disk (40GB)
• Snapshot (another 40GB potential, worst-case)
• VM swap file (1GB — sized according to RAM in theVM)

As you can see, a VM with a 40GB virtual disk can actually consume up to 81GB of space on your SAN because of a forgotten snapshot!  Use the  snapshot alarm to stay in control.  And don’t forget that VMware vSphere snapshots are perfectly suitable for production.

New Datastore Alarms

New alarms in vSphere prevent out-of-space surprises.  Administrators can monitor not only the free space on a datastore, but also the percentage overallocated — making it easy to adhere to policies concerning thin provisioning aggressiveness in your environment.

Flexible Virtual Disk Re-configuration

When creating a new VM, opting for thin-provisioned disks is as easy as checking a box.  If you change your mind later, you don’t have to start over — during a Storage VMotion operation, administrators can opt to change the VM disk format on the fly (see below).  It is also possible to inflate thin disks into thick via a new menu in the Datastore Browser.

Danger!

You’ve probably heard the now-famous quote by Tom Bittman from Gartner:

Virtualization without good management is more dangerous than not using virtualization in the first place.

That goes double for thin-provisioned virtual disks.  Without comprehensive accounting and monitoring in place, your virtual infrastructure may be heading for disaster.  This level of insight is only available with VMware vSphere 4 — and it’s built right into the platform.

What about Microsoft virtualization?  Hyper-V R2 thin provisioning — known as “dynamic disks” — is not a best practice.  Perhaps due to the lack of accounting and monitoring of storage overcommitment — especially critical now with Cluster Shared Volumes and multiple VMs per LUN.

Are you using vSphere thin provisioning?

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Easy recovery from a full VMware ESX datastore
3. Finding thin-provisioned virtual disks with PowerShell
4. PowerShell Prevents Datastore Emergencies