You can think of the VM console like a remote KVM for virtual machines  — similar to HP iLO on a physical machine.  Most of the time there are more efficient network-based remote access techniques like RDP or SSH.  However, there are some circumstances — like network misconfiguration — that absolutely require the use of a remote console.

Have you ever wondered what would happen if you were connected to a VM console while it automatically migrated with VMotion to another physical host?  If you are a long-time VMware admin, you would probably expect things to keep on working.  And that is exactly right — the console connection is seamlessly transferred to the new VMware ESX host.  This behavior is especially important when VMware DRS is dynamically moving virtual machines to optimize cluster resources.

Need to check the network settings of a VM?  No problem — you may not even notice if that VM migrates to another host while you are connected.

Here is a quick video of the situation in action:

Does Microsoft Hyper-V and System Center Virtual Machine Manager 2008 handle this scenario the same way?  Not exactly.

Related posts:

1. Hyper-V Console Disconnects During Live Migration
2. VMotion from physical ESX 4 to virtual ESX 4
3. Dear Red Hat…
4. Awesome performance through VMware DRS

• Full accounting for all provisioned storage space and complete monitoring
• Automatic response to a nearly-full datastore through vCenter Alarms, PowerShell, and Storage VMotion

You may be nearly convinced to start using thin provisioning, but still wondering…

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically grow to accommodate additional storage demand.  When VMware ESX detects this condition, virtual machines in need of additional storage are instantly paused to prevent guest operating systems from failing. Conversely, VMs that that read and write to existing allocated storage blocks will continue running without issue — not all virtual machines will be paused just because a datastore is out of space.

If you ever find yourself in this situation, it’s not hard to fix.  Here is one simple approach, step-by-step:

1. Free up some space by deleting or moving files — ISO images or powered-off VMs would be perfect
2. Resume one of the paused VMs
3. Use Storage VMotion to move the disks for that VM to another datastore
4. Resume the remaining VMs

Watch the procedure in action:

Depending on the size and storage demand of each VM, additional migrations may be needed.  An alternative resolution would be to add additional space to the SAN LUN and grow the VMFS volume.

The Experiment

To simulate a sudden storage demand by the thin-provisioned VMs in the above video, I simply copied a large file from a network share to each Windows Server 2003 VM simultaneously.

For the curious, below is a PowerShell script for the task.  Run it from anywhere — it uses Sysinternals psexec to remotely initiate a file copy on each VM from a network share.

# VM naming convention combines this string with 2-digit number
$vmPrefix = "VM2003e-"
 
# UNC path to a large file that will be copied into each VM
$iso="\\fileserver\ISO\large.iso"
 
# credentials to download above file from \\fileserver
$user="domain\username"
$pass="pa55word"
 
# pass this function a list of numbers
function createVmList ($series) {
    $vmList = @()
 
    foreach ($id in $series) {
        $vmList +=  %{"$vmPrefix{0:00}" -f $id}
    }
    $vmList
}
 
# make sure Sysinternals psexec is in your path
function psexecOnVm ($cmd, $vmList) {
    $vmList | % {Invoke-Expression "psexec.exe \\$_ -d -u $user -p $pass $cmd"}
}
 
psexecOnVM "cmd /c copy /Y $iso c:\" (createVMlist (31..39))

VMware ESX is Resilient

You may have been surprised at how easy it is to recover from a full datastore — without so much as a guest OS reboot.  It’s a testament to the rock-solid architecture behind VMware ESX and VMFS.  No other virtualization platform comes close.  Try for yourself.  See what happens if a group of thin-provisioned Hyper-V virtual machines suddenly run out of storage — it’s not going to be pretty.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. PowerShell Prevents Datastore Emergencies
3. vSphere Thin-Provisioned Disk Performance
4. Finding thin-provisioned virtual disks with PowerShell