• Full accounting for all provisioned storage space and complete monitoring
• Automatic response to a nearly-full datastore through vCenter Alarms, PowerShell, and Storage VMotion

You may be nearly convinced to start using thin provisioning, but still wondering…

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically grow to accommodate additional storage demand.  When VMware ESX detects this condition, virtual machines in need of additional storage are instantly paused to prevent guest operating systems from failing. Conversely, VMs that that read and write to existing allocated storage blocks will continue running without issue — not all virtual machines will be paused just because a datastore is out of space.

If you ever find yourself in this situation, it’s not hard to fix.  Here is one simple approach, step-by-step:

1. Free up some space by deleting or moving files — ISO images or powered-off VMs would be perfect
2. Resume one of the paused VMs
3. Use Storage VMotion to move the disks for that VM to another datastore
4. Resume the remaining VMs

Watch the procedure in action:

Depending on the size and storage demand of each VM, additional migrations may be needed.  An alternative resolution would be to add additional space to the SAN LUN and grow the VMFS volume.

The Experiment

To simulate a sudden storage demand by the thin-provisioned VMs in the above video, I simply copied a large file from a network share to each Windows Server 2003 VM simultaneously.

For the curious, below is a PowerShell script for the task.  Run it from anywhere — it uses Sysinternals psexec to remotely initiate a file copy on each VM from a network share.

# VM naming convention combines this string with 2-digit number
$vmPrefix = "VM2003e-"
 
# UNC path to a large file that will be copied into each VM
$iso="\\fileserver\ISO\large.iso"
 
# credentials to download above file from \\fileserver
$user="domain\username"
$pass="pa55word"
 
# pass this function a list of numbers
function createVmList ($series) {
    $vmList = @()
 
    foreach ($id in $series) {
        $vmList +=  %{"$vmPrefix{0:00}" -f $id}
    }
    $vmList
}
 
# make sure Sysinternals psexec is in your path
function psexecOnVm ($cmd, $vmList) {
    $vmList | % {Invoke-Expression "psexec.exe \\$_ -d -u $user -p $pass $cmd"}
}
 
psexecOnVM "cmd /c copy /Y $iso c:\" (createVMlist (31..39))

VMware ESX is Resilient

You may have been surprised at how easy it is to recover from a full datastore — without so much as a guest OS reboot.  It’s a testament to the rock-solid architecture behind VMware ESX and VMFS.  No other virtualization platform comes close.  Try for yourself.  See what happens if a group of thin-provisioned Hyper-V virtual machines suddenly run out of storage — it’s not going to be pretty.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. PowerShell Prevents Datastore Emergencies
3. vSphere Thin-Provisioned Disk Performance
4. Finding thin-provisioned virtual disks with PowerShell

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. Register a LUN full of VMs with PowerCLI
3. Responsible Thin Provisioning in VMware vSphere
4. VMware vSphere 4 has a Snapshot Alarm

Today I’d like to point out an article from ZDNet that continues the tradition with the upcoming Quick Storage Migration feature in R2:

Unlike VMWare’s storage migration feature, Storage vMotion, Microsoft has chosen not to use a “Live” storage migration approach with their Quick Storage Migration, which suspends VM activity for several minutes while the VMs are being moved, after which the VMs are resumed from suspend. Microsoft insists that this is to ensure referential integrity of the data being moved, and that the majority of large enterprise customers which they worked with to integrate this feature did not do storage migration during production hours.

Unbelievable.  Interesting use of the word “chosen.”

I’ll be sure to link back to this blog post when Hyper-V 2013 comes out and Microsoft finally has Live Storage Migration.  Stay tuned!

But seriously — are you using Storage VMotion to move production virtual machines?

Related posts:

1. How to see if Quick Migration is right for your workload
2. VMware vMotion: Over 5 Times Faster Than Hyper-V Live Migration
3. Exclusive: Hyper-V 2013 Product Roadmap
4. Hyper-V More Stable, Just as Mature as VMware vSphere?

It is an amazing thing to see an entire running VM move from one storage array to another.  Hmm, makes you wonder what the smart guys in VMware R&D will come up with next, doesn’t it?

Related posts:

1. On Quick Storage Migration in R2
2. Multiple hypervisors? What about multiple SANs?
3. Storage vendors unanimously applaud SCVMM innovation
4. The Truth About VM SAN Transfers