Virtualization management tasks such as cloning existing virtual machines or deploying new ones from templates often involve slinging around multi-gigabyte chunks of data.  It’s preferable to move or copy data on the SAN instead of the LAN because it can be faster, but more importantly, doing so reduces impact to other services that rely on the LAN.

Use the SAN, ESX!

VMware ESX will transfer VMs and templates over a SAN connection whenever possible but, if necessary, it also does a fine job of moving those bits over a standard LAN.  This goes for VMs as well as templates because with VMware vSphere, templates live on SAN datastores, too — shared among multiple ESX hosts — not on network file shares.  If templates were stored on a file server, there would be no choice other than to copy those multi-gigabyte files over the LAN and potentially impact other production traffic.

Nice design choice, VMware!  [Actually, if you go back to the early days of VirtualCenter 1.x, templates could also be stored directly on the VC server and deployed over the network.  That option was removed as of VC 2.0.]

So far, so good.  Let’s take a look at how Hyper-V stacks up.

Obfuscating the Inequalities

In the tradition of the recent storage hot add/remove claims, Microsoft has again gone the extra mile to give the appearance of feature parity.

Take a look at this excerpt from the System Center Virtual Machine Manager 2008 R2 What’s New page:

SAN migration into and out of clustered hosts: This allows virtual machines to migrate into and out of clusters using a SAN transfer, which saves the time required for copying the virtual machine file over the network.

And the features page is even more dramatic, proclaiming:

Virtual machine images can be large and difficult to move over the network. VMM auto-detects SAN infrastructure and enables copying of virtual machine images over fiber at fast speeds, thus leveraging SAN investments.

Sounds good, just like VMware — and at one-sixth the price!  But there is a small problem: it is not true.

Hyper-V uses the BITS service to transfer VMs over the LAN in almost all cases.  There is a scenario where SCVMM can orchestrate the disconnection and reconnection of a SAN LUN, “transferring” a VM to another host.  Enabling the feature requires additional configuration and some software from your SAN vendor.  Of course, it also means going back to one VM per LUN and foregoing the wonders of CSV.  Any takers?

The SCVMM Library Server

SCVMM 2008 R2 provides VM template functionality for Hyper-V.  Templates  and ISO images are stored in a Library, which uses standard Windows file services.  Obviously, that means no SAN copying for template deployments, either — kind of like VMware VirtualCenter 1.0.

Conclusion

“SAN LUN disconnect and reconnect” just doesn’t have the same ring to it as “SAN transfer” but I’m not sure that is sufficient justification for these misleading marketing claims.  And to use the word “copying” is simply dishonest.

Don’t make the mistake of assuming Hyper-V is just like VMware ESX.  Compare for yourself — seeing is believing.

Related posts:

1. IGT Part 2: SAN Misers
2. Storage vendors unanimously applaud SCVMM innovation
3. The Truth About Storage Hot Add and Remove
4. Using SCVMM to attach ISO images to VMware ESX VMs

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. Responsible Thin Provisioning in VMware vSphere
3. VI Toolkit (for Windows) 1.5 and the PowerShell prompt
4. VMware vSphere 4 has a Snapshot Alarm

Yet another nice addition to VMware vSphere 4 — making your virtual infrastructure management and monitoring as simple as possible.

Related posts:

1. Awesome performance through VMware DRS
2. Don’t know much about resource pools
3. SCVMM Damage Control
4. VM Resource Management: Hyper-V versus SCVMM

While you transition from VI3 to vSphere you will still need to keep the license server around, though.  Fortunately vCenter Server 4 is fully equipped to handle the task of pointing your ESX 3 hosts to the appropriate licensing source.

Behind the Scenes

The new architecture is a welcome change, but internally it was always very convenient to point to a central license server to quickly set up new VC and ESX systems.  Take a look at this vCenter Server licensing report — pointing to one of the internal VMware servers with one million licenses of ESX Standard.

Are you going to miss FLEXlm? I’m not.

Related posts:

1. Nice writeup on Windows Server licensing
2. Humble… but dishonest
3. If VMware ESXi 4 is so small, why is it so big?
4. The Million Hypervisor March

With template deployment comes VM (or “image”) customization — each instance of an OS needs to have a unique hostname and, in the case of Windows, a unique SID (security identifier).

Not surprisingly, each vendor has taken a different approach to image customization.  Let’s take a closer look.

VMware vCenter Server Image Customization

With VMware vSphere and vCenter Server, VM templates are simply virtual machines with a special flag that essentially prevents users from accidentally powering them on.  In fact, you can clone and customize a new VM from a standard VM if you prefer — converting to template first is not even required.  Sysprep is only run on the new VM after it is cloned — the original template/VM is never generalized by Sysprep.

The great thing about this design is that it gives you the ability to instantly convert a template back to a standard VM.  That makes it very easy to update templates and, for example, minimize the exposure of deploying virtual machines in dire need of numerous security patches.

Of course, vCenter also supports Linux guest customization, unlike competitors.

Virtual Machine Manager (SCVMM) Image Customization

Microsoft System Center Virtual Machine Manger also has VM templates, but the workflow is quite different from vCenter.

When you create a template in SCVMM, the guest OS is generalized with Sysprep before it is copied into the library.  That effectively destroys the original VM and, in fact, this is what the warning message states when you create a template.

Maintaining a clone of every VM template obviously introduces some management overhead.  Another option you might consider: when it is time to update a VM template, simply deploy a fresh VM from the previous template, make changes, and then re-create a new template from the VM.  That sounds like it will work, but there is actually a problem that prevents it from being a viable solution. For technical reasons, Sysprep can only be run on a particular OS image a total of three times.  You can read all about the details in this TechNet article. UPDATE: Activated systems in a KMS environment are not affected by this limit.

Here is what happens when you exceed the Sysprep limit when attempting to create a template in SCVMM:

Looks like SCVMM administrators will either be maintaining master clones of all templates, or reinstalling VMs from scratch when updated templates are needed.  Kind of makes you think about the whole “managing physical plus virtual” thing, doesn’t it?

Citrix XenCenter Image Customization

Curious to know how Citrix handles Windows VM cloning and customization?  They just don’t bother — the documentation simply explains how to manually run Sysprep on a Windows VM before converting to a template.  All the disadvantages of the SCVMM model — without the automation.  Nice.

Summary

VMware vCenter Server is the only virtualization management platform that offers fully functional, non-destructive VM templates for both Windows and Linux guest operating systems.

What interesting VM/template strategies have you adopted?

With apologies to the “Don’t Taze Me, Bro!” guy.

Related posts:

1. SCVMM to VMware: You will be assimilated
2. Which guest operating systems can be customized by SCVMM?
3. Even GPL can’t make Hyper-V Linux VMs well
4. Finding thin-provisioned virtual disks with PowerShell

If you happen to want a simple, built-in way to monitor your snapshots in VMware vSphere 4, look no further than this new alarm:

Very nice addition.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. Half the VM reboots on Patch Tuesday
3. Taking snapshots of VMware ESX 4 running in a VM
4. Hyper-V snapshots: not for production

Prior to vCenter Server 4, customers wishing to prune their VC database down needed to do so manually.  VMware provides an SQL script via KB 1000125 that allows administrators to delete old tasks, events, and performance stats.  By the way, I am the author of that SQL Server purge script — fear my mad T-SQL skillz!

What a difference a vSphere makes…  Now in vCenter Server 4, there is a simple administrative option to configure the retention period for tasks and events.  By default, the data is retained indefinitely, but as you can see, it’s quite easy to set it up according to your needs:

Related posts:

1. VMware vCenter really needs an agent
2. VMware vCenter Server performance stats levels
3. PowerShell Prevents Datastore Emergencies
4. VMware vCenter Server 4.0 installation video

I just took the gold vCenter Server 4.0 for a spin and rolled the camera while I was at it.  This video starts off with the installation of vCenter Server and the vSphere client, then I add an ESX 4 host and walk through the user interface a bit.

Related posts:

1. VMware DPM supported by all major server vendors
2. VMware vCenter CapacityIQ 1.0.1 for vSphere
3. VMware vCenter Server 4 task and event retention
4. vCenter for Linux article disappointing

The statistics levels are easily — perhaps too easily — configured by using the VI Client:

You can read in more depth about the performance statistics in the Basic System Administration manual and the VirtualCenter Monitoring and Performance Statistics Tech Note, which has this to say about stats level selection:

Level 1 has very low overhead on both the VirtualCenter server as well as the ESX Server hosts. Levels 2 – 4 have slightly greater overhead on ESX, but can adversely impact VirtualCenter performance if there are more than 10 ESX Server hosts. In particular Level 4 can quickly fill the VirtualCenter database, so it should only be turned on for limited periods of time.

When it says Level 4 can quickly fill the VirtualCenter database, that’s not hyperbole.  Take a look at this chart to see what happens when the “5 minute” interval is changed from level 1 to level 4:

The VPX_HIST_STAT1 table — the one that corresponds with the “5 minute” interval — immediately begins consuming many times its previous size.  After a day, the number of rows in that table increased about 20 times!  (The tick marks across the x-axis of this chart each represent an hour.)

Recovery is simple, but not immediate.  After resetting to level 1, the additional rows disappear through attrition as the rollup process does its job.

The statistics levels are configurable in order to collect more detailed performance data on your VMware infrastructure.  As with most things, there is a tradeoff — making it critical for administrators to know that a seemingly simple change via the UI can put a huge hit on your database — not only in storage requirements but performance, too.  Do yourself a favor and talk to your DBA before changing your vCenter Server performance stats levels.

Related posts:

1. VMware vCenter Server 4 task and event retention
2. VMware vCenter really needs an agent
3. New: vSphere Resource Distribution Chart
4. Upgrading your management platform from the entry-level edition

Managing the data behind these charts is actually a very demanding responsibility of vCenter: receiving, storing, and later aggregating to keep the database size under control.  Starting with VC 2.5, a couple of big changes were made in this area to help with scalability: the schema was modified to divide stats data into multiple tables, and the rollup duty (data aggregation) was delegated to the database server’s native job scheduling mechanism.  This article focuses on SQL Server and the SQL Server Agent; there is an equivalent capability in Oracle as you would suspect.

The VC 2.5 installer creates a set of three jobs (for past day, month, and week) and throws up a notice reminding you to make sure the SQL Server Agent is running on your database server — for good reason, ignore it at your own peril.  You can see the jobs in the SQL Server Management Studio.

You’ll notice in this example there are actually three instances of each of the three jobs because there are multiple vCenter databases hosted on this SQL Server.  While not typical, it makes perfect sense in a test environment.

Backstory In the initial VC 2.5 release, the jobs always had a static name, thus making it impossible to run multiple vCenter Servers against one SQL Server instance.  I knew that was going to annoy a lot of people, so I filed a bug to have it changed — finally got it in 2.5 Update 2.

What happens if the SQL Server Agent is stopped and these rollup jobs quit running?  A picture is worth a thousand words:

In this chart, the abrupt increase in database rows corresponds to the point in time that the rollup jobs were disabled.  vCenter continues receiving performance stats from hosts and dutifully depositing them in VPX_HIST_STAT1.

Essentially, the most granular data (VPX_HIST_STAT1) starts piling up, never aggregating into the summarized tables for longer-term storage.  In a large environment, it is not uncommon to see millions of rows in these tables.  If something goes wrong with the rollup jobs, those millions start multiplying in a hurry.

One nice thing about having the rollups controlled by the SQL Agent is that VC does not even need to be running for the process to take place, which also takes some load off of the VC server.  After resuming the jobs, the data flood quickly subsides and things go back on track:

One note about SQL Server Express, the bundled database included with VC: There is no SQL Agent in that product.  Before you panic, please realize that VMware engineers know this.  For those deployments, the rollup stored procedures are simply kicked off by vCenter Server (vpxd), just like VC 2.0.x.  I could not believe the creative workarounds some well-meaning folks have attempted to implement because of this situation — it had something to do with the Windows task scheduler.  Please do not do that!

Hopefully now you have a little better understanding of the relationship between your database job scheduler, vCenter Server, and the size of your vCenter database.

Related posts:

1. VMware vCenter Server 4 task and event retention
2. VMware vCenter Server performance stats levels
3. Installing vCenter on Linux Technical Preview
4. vCenter for Linux article disappointing