Recently, I was working on a project that required a large number of VMs that I had used previously but were no longer registered on a VMware ESXi host — they were just waiting patiently on a SAN LUN for their day to come.

Using the vSphere Client to connect the VMFS datastore to a host was a snap.  However, I was not enthusiastic about the several dozen clicks that would have been needed to bring them all back into the inventory for use, so I came up with a simple PowerCLI command to do all the work.

I was so pleased with the power of PowerCLI that I had to share…

Register all virtual machines on a datastore

Connect to vCenter Server with a PowerCLI session and adjust the obvious placeholders to suit your environment:

$esxhost= Get-VMHost myhost.example.com
dir 'vmstores:\myvCenterServer@443\DC1\CX4-02-LUN12\*\*.vmx' |
 % {New-VM -Host $esxhost -VMFilePath $_.DatastoreFullPath}

A few moments later, all the VMs are correctly registered and ready for action.

Build a local hosts file with PowerCLI

Another issue I encountered was the lack of DNS for guest operating systems at an offsite lab.  There was no way to connect to the various VMs by name, only IP address — not my favorite approach.  Since vSphere has information about guest networking, I quickly generated a list of IP addresses and VM names in a format suitable for pasting into my local hosts file:

Get-VM | sort-object -Property Name |
 Get-VMGuest | % {$_.ipaddress[0] + "`t" + $_.VMName}

PowerCLI to the rescue again.  What are your favorite PowerCLI one-liners?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. IGT Part 3: One-VM-per-LUN doubters
3. PowerShell Prevents Datastore Emergencies
4. VM Encapsulation

RHEV Comparison Whitepaper

Red Hat published a PDF comparing RHEV to VMware vSphere and Hyper-V.  Here is one of the entries on virtual storage:

Okay, so far so good.  Apparently, as with vSphere, virtual disks are simply files and can be stored on any of the various storage domains.  If you missed the last post on how RHEV Data Centers are limited to a single storage technology, please be sure to check it out — it’s not like vSphere.

As it turns out, the fact checkers were off the day Red Hat released that handy comparison whitepaper.  Because if you actually went through the effort to set up RHEV and create virtual machines, you would never find anything resembling a virtual disk file on the host.

Logical Volume Manager

RHEV virtual disks are actually stored in cryptically named logical volumes – no encapsulation or portability here.

Did you spot the file?  While it is common to say that “everything in UNIX is a file” this is taking that old adage to an extreme, wouldn’t you say?  Storing virtual disks in this fashion makes it nearly impossible to move VMs between hosts – even if they are powered off.  And you can forget about easily storing archive or DR copies of important VMs in an alternate location.

Even if you are running Red Hat Enteprise Linux virtual machines, the best and easiest virtualization platform to deploy, configure, and manage is VMware vSphere.  Try for yourself and see.

Related posts:

1. What is Red Hat Enterprise Virtualization?
2. Multiple hypervisors? What about multiple SANs?
3. RHEV Manager — It’s not just a clever name
4. RHEV Wrap-Up

As you may recall, Red Hat is attempting to bring their new RHEV product to market by playing the old “don’t get stuck with a single vendor” trick.  Red Hat’s CEO claims:

“…customers don’t want one platform. They want two.”

Let’s save the multi-hypervisor discussion for another day, but shift to the topic of storage diversity.

While there may be a few environments that can standardize on a single storage area network technology, it typically  makes sense to mix and match iSCSI, NFS, or Fibre Channel SANs to optimize for cost and performance.

During my recent foray into the chaotic world of Red Hat Enterprise Virtualization, I encountered an unbelievable storage limitation with their new KVM hypervisor.

“Virtualization diversity… when and where we say”

VMware vSphere customers are free to mix and match supported storage technologies within datacenters, clusters, and even hosts.

I would have thought that a company like Red Hat that is pushing heterogeneous solutions and multiple vendors might feel the same way when it comes to storage.  But that’s not quite how things go in the land of Red Hat Enterprise Virtualization.

With RHEV, a “Data Center” is configured to use a single type of storage;  all clusters and hosts in that Data Center are restricted to that, so choose wisely:

Storage Freedom

VMware vSphere allows administrators to move virtual disks for running VMs from one array to another — even between different types of arrays — with zero downtime.  Storage vMotion is a very powerful feature and it is simply not available from Red Hat – only VMware.

The Best Choice: VMware vSphere

Virtual machines have differing storage requirements — certainly not a one-size-fits-all component of an efficient virtual infrastructure.  The best platform for building your own private cloud is also the one with the broadest support for today’s storage technologies:  VMware vSphere.

Oh, by the way, did you know that VMware is positioned in the Leaders Quadrant of Gartner’s newly-released x86 Server Virtualization Magic Quadrant.  Take a look at the full article — I’ll give a free subscription to the VCritical RSS feed to the first 10 people that spot Red Hat’s position.

Related posts:

1. What is Red Hat Enterprise Virtualization?
2. These are not the files you are looking for
3. Red Hat Enterprise Linux is not Enterprise Virtualization
4. RHEV Wrap-Up

Virtualization management tasks such as cloning existing virtual machines or deploying new ones from templates often involve slinging around multi-gigabyte chunks of data.  It’s preferable to move or copy data on the SAN instead of the LAN because it can be faster, but more importantly, doing so reduces impact to other services that rely on the LAN.

Use the SAN, ESX!

VMware ESX will transfer VMs and templates over a SAN connection whenever possible but, if necessary, it also does a fine job of moving those bits over a standard LAN.  This goes for VMs as well as templates because with VMware vSphere, templates live on SAN datastores, too — shared among multiple ESX hosts — not on network file shares.  If templates were stored on a file server, there would be no choice other than to copy those multi-gigabyte files over the LAN and potentially impact other production traffic.

Nice design choice, VMware!  [Actually, if you go back to the early days of VirtualCenter 1.x, templates could also be stored directly on the VC server and deployed over the network.  That option was removed as of VC 2.0.]

So far, so good.  Let’s take a look at how Hyper-V stacks up.

Obfuscating the Inequalities

In the tradition of the recent storage hot add/remove claims, Microsoft has again gone the extra mile to give the appearance of feature parity.

Take a look at this excerpt from the System Center Virtual Machine Manager 2008 R2 What’s New page:

SAN migration into and out of clustered hosts: This allows virtual machines to migrate into and out of clusters using a SAN transfer, which saves the time required for copying the virtual machine file over the network.

And the features page is even more dramatic, proclaiming:

Virtual machine images can be large and difficult to move over the network. VMM auto-detects SAN infrastructure and enables copying of virtual machine images over fiber at fast speeds, thus leveraging SAN investments.

Sounds good, just like VMware — and at one-sixth the price!  But there is a small problem: it is not true.

Hyper-V uses the BITS service to transfer VMs over the LAN in almost all cases.  There is a scenario where SCVMM can orchestrate the disconnection and reconnection of a SAN LUN, “transferring” a VM to another host.  Enabling the feature requires additional configuration and some software from your SAN vendor.  Of course, it also means going back to one VM per LUN and foregoing the wonders of CSV.  Any takers?

The SCVMM Library Server

SCVMM 2008 R2 provides VM template functionality for Hyper-V.  Templates  and ISO images are stored in a Library, which uses standard Windows file services.  Obviously, that means no SAN copying for template deployments, either — kind of like VMware VirtualCenter 1.0.

Conclusion

“SAN LUN disconnect and reconnect” just doesn’t have the same ring to it as “SAN transfer” but I’m not sure that is sufficient justification for these misleading marketing claims.  And to use the word “copying” is simply dishonest.

Don’t make the mistake of assuming Hyper-V is just like VMware ESX.  Compare for yourself — seeing is believing.

Related posts:

1. IGT Part 2: SAN Misers
2. Storage vendors unanimously applaud SCVMM innovation
3. The Truth About Storage Hot Add and Remove
4. Using SCVMM to attach ISO images to VMware ESX VMs

• vStorage VMFS Volume Grow
• Hot Extend for Virtual Disks
• Hot Virtual Disk Add/Remove

With these capabilities, if space gets tight in your vSphere environment, it is easy to be proactive and address the issue before anyone notices.  The process goes something like this:

• Allocate additional physical hard disk space on your SAN to an appropriate LUN
• Grow your VMFS datastore onto the newly added free space
• Extend, or add new, virtual disks for the VMs that need more storage
• Expand the volume inside the guest to create more usable space

All without a reboot: zero downtime.

Not All Virtualization Platforms are Created Equal

Don’t jump to the conclusion that all hypervisors offer the same flexibility.  Perhaps you are wondering about Hyper-V capabilities?

First, let’s take a look at some Microsoft System Center Virtual Machine Manager 2008 R2 (SCVMM) marketing statements:

The What’s New page announces:

Hot addition/removal of Storage: Allows the addition and removal of storage to virtualized infrastructure without interruption. Additionally, “live” management of virtual hard disk (VHDs) or iSCSI pass through disks, allows administrators to take advantage of additional backup scenarios and readily use mission critical and storage-intensive applications.

The Top Benefits list proclaims:

Hot addition/removal of storage: With this capability, administrators can quickly and efficiently respond to changing storage requirements of virtual machines. This ability to hot-add additional storage eliminates the previous need to take the host down to upgrade storage thus increasing business continuity for end users and reducing complexity for administrators. Additionally it allows administrators to confidently deploy mission critical applications (in which up-time is of paramount importance) that may have rapidly changing storage requirements such as web, database or other business applications.

An IT decision-maker just might get the impression that both ESX and Hyper-V have essentially the same features. They do not.

You may be surprised to find out that all of the descriptions above merely refer to adding a new virtual disk to a VM — providing the conditions are right.  You cannot grow an existing VHD, and you can’t safely remove a VHD with SCVMM.

Plan Ahead

First things first.  If your Hyper-V VM does not have a virtual SCSI adapter — templates and VMs from Hyper-V R1 do not — you won’t be able to hot add a new VHD until you correct that shortcoming.  Hello downtime.

SCVMM can add a new blank virtual disk to a VM or it can copy an existing one across the network from the Library — if you copied it there beforehand, but there is no way to add a VHD that may already be present on your SAN — even if it is already sitting right next to the destination VM.

Removal?

Up-time may be of paramount importance, but preventing data-loss was evidently not part of the original design.  Removing a VHD with SCVMM results in the immediate deletion of the underlying VHD file.  Ouch!  Thankfully, a recent patch improves administrator job security by throwing up a warning before this happens, providing an option to cancel.  There is no way to simply disconnect a VHD using SCVMM.

It turns out that if you really want to take advantage of those “Additional Backup Scenarios” by hot adding and removing virtual disks, you need your trusty Hyper-V Manager utility.  Still think System Center is a single pane of glass?  And that’s not the only task that requires administrators to switch between Hyper-V interfaces.

Conclusion

Microsoft is trying hard to ride the coattails of VMware ESX.  The latest release of Hyper-V R2 still does not have capabilities enjoyed by VMware administrators since the ESX 3.5 days.

Don’t believe the obfuscated marketing literature.  VMware vSphere is for real.

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Responsible Thin Provisioning in VMware vSphere
3. The Truth About VM SAN Transfers
4. Easy recovery from a full VMware ESX datastore

Under the supervision of VMware vCenter Server, ESX 4 thin provisioning is safe and reliable even for production workloads thanks to advanced storage accounting and built-in monitoring.  And even if the worst does happen — an unexpectedly full datastore — recovery is simple.

Now that you are convinced that VMware vSphere thin provisioning has a place in your data center, you may be wondering about performance tradeoffs.

It turns out that thin disks perform just about as well as thick disks.

New Performance Results

In this brand-new VMware vStorage Thin Provisioning performance study, VMware performance engineers compare thick and thin disk performance.  This must-read document covers several important topics, such as:

• I/O-intensive benchmarking with a 16-node ESX cluster
• File copy benchmarking with 2 ESX hosts
• Fragmentation impact
• Thin provisioning affect on co-located thick disks

Now, go forth and provision… thinly.

Related posts:

1. Finding thin-provisioned virtual disks with PowerShell
2. Responsible Thin Provisioning in VMware vSphere
3. Easy recovery from a full VMware ESX datastore
4. The Truth About Storage Hot Add and Remove

• Full accounting for all provisioned storage space and complete monitoring
• Automatic response to a nearly-full datastore through vCenter Alarms, PowerShell, and Storage VMotion

You may be nearly convinced to start using thin provisioning, but still wondering…

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically grow to accommodate additional storage demand.  When VMware ESX detects this condition, virtual machines in need of additional storage are instantly paused to prevent guest operating systems from failing. Conversely, VMs that that read and write to existing allocated storage blocks will continue running without issue — not all virtual machines will be paused just because a datastore is out of space.

If you ever find yourself in this situation, it’s not hard to fix.  Here is one simple approach, step-by-step:

1. Free up some space by deleting or moving files — ISO images or powered-off VMs would be perfect
2. Resume one of the paused VMs
3. Use Storage VMotion to move the disks for that VM to another datastore
4. Resume the remaining VMs

Watch the procedure in action:

Depending on the size and storage demand of each VM, additional migrations may be needed.  An alternative resolution would be to add additional space to the SAN LUN and grow the VMFS volume.

The Experiment

To simulate a sudden storage demand by the thin-provisioned VMs in the above video, I simply copied a large file from a network share to each Windows Server 2003 VM simultaneously.

For the curious, below is a PowerShell script for the task.  Run it from anywhere — it uses Sysinternals psexec to remotely initiate a file copy on each VM from a network share.

# VM naming convention combines this string with 2-digit number
$vmPrefix = "VM2003e-"
 
# UNC path to a large file that will be copied into each VM
$iso="\\fileserver\ISO\large.iso"
 
# credentials to download above file from \\fileserver
$user="domain\username"
$pass="pa55word"
 
# pass this function a list of numbers
function createVmList ($series) {
    $vmList = @()
 
    foreach ($id in $series) {
        $vmList +=  %{"$vmPrefix{0:00}" -f $id}
    }
    $vmList
}
 
# make sure Sysinternals psexec is in your path
function psexecOnVm ($cmd, $vmList) {
    $vmList | % {Invoke-Expression "psexec.exe \\$_ -d -u $user -p $pass $cmd"}
}
 
psexecOnVM "cmd /c copy /Y $iso c:\" (createVMlist (31..39))

VMware ESX is Resilient

You may have been surprised at how easy it is to recover from a full datastore — without so much as a guest OS reboot.  It’s a testament to the rock-solid architecture behind VMware ESX and VMFS.  No other virtualization platform comes close.  Try for yourself.  See what happens if a group of thin-provisioned Hyper-V virtual machines suddenly run out of storage — it’s not going to be pretty.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. vSphere Thin-Provisioned Disk Performance
3. PowerShell Prevents Datastore Emergencies
4. Finding thin-provisioned virtual disks with PowerShell

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. Register a LUN full of VMs with PowerCLI
3. Responsible Thin Provisioning in VMware vSphere
4. VI Toolkit (for Windows) 1.5 and the PowerShell prompt

What’s new is not necessarily the technology — it’s the management.  In fact, veteran VMware ESX admins have been creating thin provisioned virtual disks for years — for controlled scenarios — by way of the vmkfstools command.

Before vSphere and ESX 4, however, thin disks came with risk — there was no simple way of accounting for the overcommitted storage on each LUN.  Even with multiple gigabytes of free space, a small gang of thin-provisioned virtual machines could quickly quickly grow to exceed datastore capacity during a sudden demand spike.

Complete Storage Accounting

Now in vSphere 4 there is a new element in the capacity section of the datastore summary tab that shows total provisioned space — the maximum potential growth of all virtual machines if thin provisioned disks were fully utilized:

Virtual machines with snapshots have the potential of consuming even more datastore space, so vSphere accounts for this condition, too.  Take a look at this VM Summary tab, where the total provisioned storage includes:

• Virtual hard disk (40GB)
• Snapshot (another 40GB potential, worst-case)
• VM swap file (1GB — sized according to RAM in theVM)

As you can see, a VM with a 40GB virtual disk can actually consume up to 81GB of space on your SAN because of a forgotten snapshot!  Use the  snapshot alarm to stay in control.  And don’t forget that VMware vSphere snapshots are perfectly suitable for production.

New Datastore Alarms

New alarms in vSphere prevent out-of-space surprises.  Administrators can monitor not only the free space on a datastore, but also the percentage overallocated — making it easy to adhere to policies concerning thin provisioning aggressiveness in your environment.

Flexible Virtual Disk Re-configuration

When creating a new VM, opting for thin-provisioned disks is as easy as checking a box.  If you change your mind later, you don’t have to start over — during a Storage VMotion operation, administrators can opt to change the VM disk format on the fly (see below).  It is also possible to inflate thin disks into thick via a new menu in the Datastore Browser.

Danger!

You’ve probably heard the now-famous quote by Tom Bittman from Gartner:

Virtualization without good management is more dangerous than not using virtualization in the first place.

That goes double for thin-provisioned virtual disks.  Without comprehensive accounting and monitoring in place, your virtual infrastructure may be heading for disaster.  This level of insight is only available with VMware vSphere 4 — and it’s built right into the platform.

What about Microsoft virtualization?  Hyper-V R2 thin provisioning — known as “dynamic disks” — is not a best practice.  Perhaps due to the lack of accounting and monitoring of storage overcommitment — especially critical now with Cluster Shared Volumes and multiple VMs per LUN.

Are you using vSphere thin provisioning?

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Easy recovery from a full VMware ESX datastore
3. Finding thin-provisioned virtual disks with PowerShell
4. VMware vSphere 4 has a Snapshot Alarm

If you happen to want a simple, built-in way to monitor your snapshots in VMware vSphere 4, look no further than this new alarm:

Very nice addition.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. Half the VM reboots on Patch Tuesday
3. Taking snapshots of VMware ESX 4 running in a VM
4. Hyper-V snapshots: not for production