Virtualization management tasks such as cloning existing virtual machines or deploying new ones from templates often involve slinging around multi-gigabyte chunks of data.  It’s preferable to move or copy data on the SAN instead of the LAN because it can be faster, but more importantly, doing so reduces impact to other services that rely on the LAN.

Use the SAN, ESX!

VMware ESX will transfer VMs and templates over a SAN connection whenever possible but, if necessary, it also does a fine job of moving those bits over a standard LAN.  This goes for VMs as well as templates because with VMware vSphere, templates live on SAN datastores, too — shared among multiple ESX hosts — not on network file shares.  If templates were stored on a file server, there would be no choice other than to copy those multi-gigabyte files over the LAN and potentially impact other production traffic.

Nice design choice, VMware!  [Actually, if you go back to the early days of VirtualCenter 1.x, templates could also be stored directly on the VC server and deployed over the network.  That option was removed as of VC 2.0.]

So far, so good.  Let’s take a look at how Hyper-V stacks up.

Obfuscating the Inequalities

In the tradition of the recent storage hot add/remove claims, Microsoft has again gone the extra mile to give the appearance of feature parity.

Take a look at this excerpt from the System Center Virtual Machine Manager 2008 R2 What’s New page:

SAN migration into and out of clustered hosts: This allows virtual machines to migrate into and out of clusters using a SAN transfer, which saves the time required for copying the virtual machine file over the network.

And the features page is even more dramatic, proclaiming:

Virtual machine images can be large and difficult to move over the network. VMM auto-detects SAN infrastructure and enables copying of virtual machine images over fiber at fast speeds, thus leveraging SAN investments.

Sounds good, just like VMware — and at one-sixth the price!  But there is a small problem: it is not true.

Hyper-V uses the BITS service to transfer VMs over the LAN in almost all cases.  There is a scenario where SCVMM can orchestrate the disconnection and reconnection of a SAN LUN, “transferring” a VM to another host.  Enabling the feature requires additional configuration and some software from your SAN vendor.  Of course, it also means going back to one VM per LUN and foregoing the wonders of CSV.  Any takers?

The SCVMM Library Server

SCVMM 2008 R2 provides VM template functionality for Hyper-V.  Templates  and ISO images are stored in a Library, which uses standard Windows file services.  Obviously, that means no SAN copying for template deployments, either — kind of like VMware VirtualCenter 1.0.

Conclusion

“SAN LUN disconnect and reconnect” just doesn’t have the same ring to it as “SAN transfer” but I’m not sure that is sufficient justification for these misleading marketing claims.  And to use the word “copying” is simply dishonest.

Don’t make the mistake of assuming Hyper-V is just like VMware ESX.  Compare for yourself — seeing is believing.

Related posts:

1. IGT Part 2: SAN Misers
2. Storage vendors unanimously applaud SCVMM innovation
3. The Truth About Storage Hot Add and Remove
4. Using SCVMM to attach ISO images to VMware ESX VMs

• vStorage VMFS Volume Grow
• Hot Extend for Virtual Disks
• Hot Virtual Disk Add/Remove

With these capabilities, if space gets tight in your vSphere environment, it is easy to be proactive and address the issue before anyone notices.  The process goes something like this:

• Allocate additional physical hard disk space on your SAN to an appropriate LUN
• Grow your VMFS datastore onto the newly added free space
• Extend, or add new, virtual disks for the VMs that need more storage
• Expand the volume inside the guest to create more usable space

All without a reboot: zero downtime.

Not All Virtualization Platforms are Created Equal

Don’t jump to the conclusion that all hypervisors offer the same flexibility.  Perhaps you are wondering about Hyper-V capabilities?

First, let’s take a look at some Microsoft System Center Virtual Machine Manager 2008 R2 (SCVMM) marketing statements:

The What’s New page announces:

Hot addition/removal of Storage: Allows the addition and removal of storage to virtualized infrastructure without interruption. Additionally, “live” management of virtual hard disk (VHDs) or iSCSI pass through disks, allows administrators to take advantage of additional backup scenarios and readily use mission critical and storage-intensive applications.

The Top Benefits list proclaims:

Hot addition/removal of storage: With this capability, administrators can quickly and efficiently respond to changing storage requirements of virtual machines. This ability to hot-add additional storage eliminates the previous need to take the host down to upgrade storage thus increasing business continuity for end users and reducing complexity for administrators. Additionally it allows administrators to confidently deploy mission critical applications (in which up-time is of paramount importance) that may have rapidly changing storage requirements such as web, database or other business applications.

An IT decision-maker just might get the impression that both ESX and Hyper-V have essentially the same features. They do not.

You may be surprised to find out that all of the descriptions above merely refer to adding a new virtual disk to a VM — providing the conditions are right.  You cannot grow an existing VHD, and you can’t safely remove a VHD with SCVMM.

Plan Ahead

First things first.  If your Hyper-V VM does not have a virtual SCSI adapter — templates and VMs from Hyper-V R1 do not — you won’t be able to hot add a new VHD until you correct that shortcoming.  Hello downtime.

SCVMM can add a new blank virtual disk to a VM or it can copy an existing one across the network from the Library — if you copied it there beforehand, but there is no way to add a VHD that may already be present on your SAN — even if it is already sitting right next to the destination VM.

Removal?

Up-time may be of paramount importance, but preventing data-loss was evidently not part of the original design.  Removing a VHD with SCVMM results in the immediate deletion of the underlying VHD file.  Ouch!  Thankfully, a recent patch improves administrator job security by throwing up a warning before this happens, providing an option to cancel.  There is no way to simply disconnect a VHD using SCVMM.

It turns out that if you really want to take advantage of those “Additional Backup Scenarios” by hot adding and removing virtual disks, you need your trusty Hyper-V Manager utility.  Still think System Center is a single pane of glass?  And that’s not the only task that requires administrators to switch between Hyper-V interfaces.

Conclusion

Microsoft is trying hard to ride the coattails of VMware ESX.  The latest release of Hyper-V R2 still does not have capabilities enjoyed by VMware administrators since the ESX 3.5 days.

Don’t believe the obfuscated marketing literature.  VMware vSphere is for real.

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Responsible Thin Provisioning in VMware vSphere
3. The Truth About VM SAN Transfers
4. Easy recovery from a full VMware ESX datastore

Under the supervision of VMware vCenter Server, ESX 4 thin provisioning is safe and reliable even for production workloads thanks to advanced storage accounting and built-in monitoring.  And even if the worst does happen — an unexpectedly full datastore — recovery is simple.

Now that you are convinced that VMware vSphere thin provisioning has a place in your data center, you may be wondering about performance tradeoffs.

It turns out that thin disks perform just about as well as thick disks.

New Performance Results

In this brand-new VMware vStorage Thin Provisioning performance study, VMware performance engineers compare thick and thin disk performance.  This must-read document covers several important topics, such as:

• I/O-intensive benchmarking with a 16-node ESX cluster
• File copy benchmarking with 2 ESX hosts
• Fragmentation impact
• Thin provisioning affect on co-located thick disks

Now, go forth and provision… thinly.

Related posts:

1. Finding thin-provisioned virtual disks with PowerShell
2. Responsible Thin Provisioning in VMware vSphere
3. Easy recovery from a full VMware ESX datastore
4. The Truth About Storage Hot Add and Remove

• Full accounting for all provisioned storage space and complete monitoring
• Automatic response to a nearly-full datastore through vCenter Alarms, PowerShell, and Storage VMotion

You may be nearly convinced to start using thin provisioning, but still wondering…

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically grow to accommodate additional storage demand.  When VMware ESX detects this condition, virtual machines in need of additional storage are instantly paused to prevent guest operating systems from failing. Conversely, VMs that that read and write to existing allocated storage blocks will continue running without issue — not all virtual machines will be paused just because a datastore is out of space.

If you ever find yourself in this situation, it’s not hard to fix.  Here is one simple approach, step-by-step:

1. Free up some space by deleting or moving files — ISO images or powered-off VMs would be perfect
2. Resume one of the paused VMs
3. Use Storage VMotion to move the disks for that VM to another datastore
4. Resume the remaining VMs

Watch the procedure in action:

Depending on the size and storage demand of each VM, additional migrations may be needed.  An alternative resolution would be to add additional space to the SAN LUN and grow the VMFS volume.

The Experiment

To simulate a sudden storage demand by the thin-provisioned VMs in the above video, I simply copied a large file from a network share to each Windows Server 2003 VM simultaneously.

For the curious, below is a PowerShell script for the task.  Run it from anywhere — it uses Sysinternals psexec to remotely initiate a file copy on each VM from a network share.

# VM naming convention combines this string with 2-digit number
$vmPrefix = "VM2003e-"
 
# UNC path to a large file that will be copied into each VM
$iso="\\fileserver\ISO\large.iso"
 
# credentials to download above file from \\fileserver
$user="domain\username"
$pass="pa55word"
 
# pass this function a list of numbers
function createVmList ($series) {
    $vmList = @()
 
    foreach ($id in $series) {
        $vmList +=  %{"$vmPrefix{0:00}" -f $id}
    }
    $vmList
}
 
# make sure Sysinternals psexec is in your path
function psexecOnVm ($cmd, $vmList) {
    $vmList | % {Invoke-Expression "psexec.exe \\$_ -d -u $user -p $pass $cmd"}
}
 
psexecOnVM "cmd /c copy /Y $iso c:\" (createVMlist (31..39))

VMware ESX is Resilient

You may have been surprised at how easy it is to recover from a full datastore — without so much as a guest OS reboot.  It’s a testament to the rock-solid architecture behind VMware ESX and VMFS.  No other virtualization platform comes close.  Try for yourself.  See what happens if a group of thin-provisioned Hyper-V virtual machines suddenly run out of storage — it’s not going to be pretty.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. vSphere Thin-Provisioned Disk Performance
3. PowerShell Prevents Datastore Emergencies
4. Finding thin-provisioned virtual disks with PowerShell

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. Responsible Thin Provisioning in VMware vSphere
3. VI Toolkit (for Windows) 1.5 and the PowerShell prompt
4. VMware vSphere 4 has a Snapshot Alarm

What’s new is not necessarily the technology — it’s the management.  In fact, veteran VMware ESX admins have been creating thin provisioned virtual disks for years — for controlled scenarios — by way of the vmkfstools command.

Before vSphere and ESX 4, however, thin disks came with risk — there was no simple way of accounting for the overcommitted storage on each LUN.  Even with multiple gigabytes of free space, a small gang of thin-provisioned virtual machines could quickly quickly grow to exceed datastore capacity during a sudden demand spike.

Complete Storage Accounting

Now in vSphere 4 there is a new element in the capacity section of the datastore summary tab that shows total provisioned space — the maximum potential growth of all virtual machines if thin provisioned disks were fully utilized:

Virtual machines with snapshots have the potential of consuming even more datastore space, so vSphere accounts for this condition, too.  Take a look at this VM Summary tab, where the total provisioned storage includes:

• Virtual hard disk (40GB)
• Snapshot (another 40GB potential, worst-case)
• VM swap file (1GB — sized according to RAM in theVM)

As you can see, a VM with a 40GB virtual disk can actually consume up to 81GB of space on your SAN because of a forgotten snapshot!  Use the  snapshot alarm to stay in control.  And don’t forget that VMware vSphere snapshots are perfectly suitable for production.

New Datastore Alarms

New alarms in vSphere prevent out-of-space surprises.  Administrators can monitor not only the free space on a datastore, but also the percentage overallocated — making it easy to adhere to policies concerning thin provisioning aggressiveness in your environment.

Flexible Virtual Disk Re-configuration

When creating a new VM, opting for thin-provisioned disks is as easy as checking a box.  If you change your mind later, you don’t have to start over — during a Storage VMotion operation, administrators can opt to change the VM disk format on the fly (see below).  It is also possible to inflate thin disks into thick via a new menu in the Datastore Browser.

Danger!

You’ve probably heard the now-famous quote by Tom Bittman from Gartner:

Virtualization without good management is more dangerous than not using virtualization in the first place.

That goes double for thin-provisioned virtual disks.  Without comprehensive accounting and monitoring in place, your virtual infrastructure may be heading for disaster.  This level of insight is only available with VMware vSphere 4 — and it’s built right into the platform.

What about Microsoft virtualization?  Hyper-V R2 thin provisioning — known as “dynamic disks” — is not a best practice.  Perhaps due to the lack of accounting and monitoring of storage overcommitment — especially critical now with Cluster Shared Volumes and multiple VMs per LUN.

Are you using vSphere thin provisioning?

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Easy recovery from a full VMware ESX datastore
3. Finding thin-provisioned virtual disks with PowerShell
4. VMware vSphere 4 has a Snapshot Alarm

If you happen to want a simple, built-in way to monitor your snapshots in VMware vSphere 4, look no further than this new alarm:

Very nice addition.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. Half the VM reboots on Patch Tuesday
3. Taking snapshots of VMware ESX 4 running in a VM
4. Hyper-V snapshots: not for production

The VI Client interface will not let administrators create new virtual machines with thin-provisioned disks.  Even so, it is entirely possible that your environment contains such disks.  Let’s take a look at a couple of ways to create these disks and then use a VI Toolkit (for Windows) PowerShell script to see if any exist.

Manual VMDK Creation

One way that thin disks may be introduced into an environment is by manually creating a VMDK file and then attaching that file to a virtual machine.  This can be accomplished by the vmkfstools command with the -d thin option.  This procedure is not for beginners, so if you have used it you presumably evaluated the risks of overcommitting your storage.  But what if one of your colleagues did this without telling you?

Compact Clones

Another source of thin disks requires a little background first:

There is a prominent option in the VI Client interface to convert a virtual machine to template (and vice versa).  But if you right-click a VM in the inventory, you are presented with a variation of that feature:  Clone to Template.

The wizard that launches in response to that option allows an administrator to optionally choose the Compact disk format — this is a thin-provisioned disk.  If your template has a 50GB virtual disk but only 8GB are actually used, this feature can provide significant storage savings.

So far so good.  Now, what do you suppose happens if you decide to convert that template to a VM?  Nothing happens — it just works.  You are free to use that VM just like any other in your inventory.  But this VM keeps its thin-provisioned disk, a fact that is not discernible through the user interface.

Find the Thin-Provisioned Disks

I wrote a small VI Toolkit (for Windows) PowerShell script that identifies latent thin disks in both VMs and templates.  I hope you find it useful:

# FindThinDisks.ps1
#
# Identifies VMs and templates that are using thin-provisioned
# virtual disks.
 
# Version 1.0  January 14, 2009
# Eric Gray
 
$vmtp = Get-VM
$vmtp += Get-Template
 
foreach($vm in $vmtp | Get-View){
  foreach($dev in $vm.Config.Hardware.Device){
    if(($dev.GetType()).Name -eq "VirtualDisk"){
      if($dev.Backing.ThinProvisioned -eq $true) {
        $vm.Name + "`t" + $dev.Backing.FileName
      }
    }
  }
}

Thin-provisioning is a great tool to have in your toolbox, when used properly.

Script inspired by this thread.

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. PowerShell Prevents Datastore Emergencies
3. Responsible Thin Provisioning in VMware vSphere
4. Easy recovery from a full VMware ESX datastore

If VMware vCenter and SCVMM use opposing techniques to perform this operation, what happens when an administrator tries to use SCVMM to manage ESX or ESXi?  Evidently not wanting to give ESX special treatment, SCVMM copies the ISO just like it does for Hyper-V.

“Copies it how?” you must be thinking.  With SFTP through the ESX service console, of course!

Let’s review the numerous disadvantages of not having a shared ISO image repository:

1. Takes longer to mount an ISO to a VM
2. Ties up expensive SAN storage for duplicate copies of images
3. Uses network bandwidth to copy these multi-gigabyte files
4. Wastes ESX service console CPU cycles to receive SFTP uploads

I was giving a presentation this morning to the VMware Sales team and a question was raised by one of the participants:  If you are using VMware ESXi how is the image transferred — there is no service console and therefore no SSH daemon.  Not having tried this operation, I theorized that it would be done using the API — similar to how the VI Client can easily upload files to an ESXi datastore.

After that presentation, I hijacked a colleague’s ESXi system and tried to connect an ISO image to a VM.  Sure enough, the file was transferred via HTTP.  So far, so good.

But there was a problem.  Shortly after this job finished, the VM in question started showing warning messages about not being able to locate the file.  I opened the VI Client and used the datastore browser to look at the VM directory.  There was no ISO image to be found.

Digging a little deeper, I grabbed the hostd.log file from the ESXi server to look for clues.   Here is an excerpt:

 warning] Exception while processing request: Input/output error
 info] Incomplete message body received for PUT /folder/win2k3r2/en_win_srv_

Looks like a problem with the SCVMM/ESXi integration.  As far as I can tell the host is properly configured in SCVMM — which is accomplished by providing the root password so tasks like this can be performed.  (But that’s an entirely different issue to talk about.)  I separately verified that I could upload a file to that same host using the VI Client.

I must conclude that it is not possible to use SCVMM to attach an ISO image file to VMware ESXi virtual machines.  Is SCVMM a single pane of glass to manage your entire virtual infrastructure?  I don’t think so.

Related posts:

1. Storage vendors unanimously applaud SCVMM innovation
2. IGT Part 2: SAN Misers
3. If VMware ESXi 4 is so small, why is it so big?
4. Managing VI3 with SCVMM considered harmful

Snapshots are one of the great features of virtualization, but forget about one on an active VM for too long and you may be calling for cleanup on aisle one.  Recently there was a fair amount of discussion among VM bloggers about snapshots.  Jason Boche had a nice article that summarized some monitoring strategies and Rich Brambley also weighed in.

You may be wondering what makes snapshots so great.  At the top of the list is the ability to have an easy roll-back plan for failed upgrades.  For example, an administrator can take a VM snapshot before applying patches to a guest OS.  After rebooting and testing, if all is well, simply delete the snapshot and move on — the changes are merged while the virtual machine is running.  If for some reason a patch caused problems, the previous state can be quickly restored.

That’s how VMware ESX works.  Is Microsoft Hyper-V the same?

It turns out that Hyper-V snapshots (or SCVMM checkpoints, depending on which single pane of glass you are using) use a slightly different design. The Microsoft interfaces allow administrators to remove snapshots from powered-on VMs, and after doing so, they appear to be gone:

But, upon closer inspection, the changes are not actually merged until the VM is powered off:

What happens if an administrator is not aware of this behavior?  Just as if the snapshot was never deleted, it grow and grow until the LUN on which it lives runs out of space:

And when that happens, the VM will go into a Paused-Critical state:

Cleanup on aisle one!

What is the best way to recover gracefully from such a problem, if there really is no free space on the LUN? Gabes Virtual World published a great tip that could help next time.  Maybe all that ISO file copying isn’t so bad after all.

Related posts:

1. Hyper-V snapshots: not for production
2. Half the VM reboots on Patch Tuesday
3. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off
4. Checkshots or Snappoints?