However, one limitation that some have discovered while using this lab-testing technique is the lack of ability to use snapshots with virtual ESX systems.  In fact, after taking a snapshot of a virtual ESX VM, you will see the system boot into the recovery shell like so:

Ouch.

Can we fix it?

I asked some super-smart engineers inside VMware about the issue, and of course they know all about it and how to make it work.

The solution is to enable an undocumented advanced configuration option on the physical ESX host like so:

In other words:

• Log into the console of the physical VMware ESX 4 host
• Run the following command: esxcfg-advcfg -s 1  /COW/PermitVmfsOnRedoHierarchy
• Verify the setting with the “get” flag: esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy

Do I even need to say that this stuff is not supported by VMware Global Support Services (GSS)?  Well, I just did.

Yes we can!

After executing that one command on your physical machine, take a snapshot and boot up your virtual ESX:

Everything works just like you need it to now — you can even create a snapshot hierarchy:

Now what?

Were you constrained by the lack of virtual ESX snapshots?  What interesting things will you use this for?

Related posts:

1. Half the VM reboots on Patch Tuesday
2. VMware vSphere 4 has a Snapshot Alarm
3. Hyper-V snapshots: not for production
4. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off

One outstanding achievement is the way the Hyper-V architects managed to completely collapse the guest operating system layer directly into the hypervisor.  Take a look at this diagram from a Microsoft Virtualization competitive brochure:

[Yes, it's that same brochure again.]

There are several key differences between Microsoft and VMware hypervisor architectures.  While the hardware and applications layers are pretty straightforward, applications don’t run directly on hardware.  As you are about to see, these two platforms take very different approaches when it comes to virtualizing workloads.

The VMware vSphere solution is to use a thin, high-performance virtualization layer that enables a single physical system to simultaneously host multiple virtual machines.  Each of these virtual machines executes isolated instances of traditional guest operating systems — Windows, Linux, Solaris, Netware, etc.

Due to apparent technological advancements in Hyper-V, it is now possible to run applications of all kinds directly on that integrated Windows OS/hypervisor layer without the need for a guest operating system at all.  This allows Microsoft to omit an entire layer from their virtualization architecture.

I’m sure you have seen various announcements about the exceptional support that Hyper-V has for Linux workloads.  Without a guest operating system layer, one must logically conclude that Linux applications are actually running on the Windows/Hyper-V hypervisor integration directly.  Spectacular!

I thought I’d test this technological breakthrough in the lab myself by running Postfix, a very popular Linux SMTP mail server, on Windows Server 2008 R2 with Hyper-V.  According to the diagram above, Postfix is classified as an application — the third layer.

Now, in order to install Postfix, I have to interact with the second layer in the stack, which is the “Microsoft Operating System w/Virtualization.”  Since I’m an RHCE, I decided to use a Red Hat Enterprise Linux package of Postfix, but we can all agree that SUSE Linux Enterprise Server 11 would work equally well.  However, I would not recommend using Ubuntu packages since that distribution is not supported by Hyper-V.  Another side benefit to this technique is the fact that Windows has great mouse support.

Just as one would do on a legacy Linux system, I opened a trusty command shell and installed the package like so:

This is the first step to getting Postfix installed and running.  I do have to admit that it took a few additional troubleshooting steps in my case — most certainly unique to my environment, so I will spare the details.

In order to complete the Postfix implementation, you’ll need to edit several configuration files.  Do not use Notepad for this, as it does not understand the peculiar format of these text files:

One would think that the virtualization layer could also dynamically accommodate for UNIX line formatting, but that evidently was not part of the design.  Fortunately, this is a very minor issue since WordPad can be used without problem:

Once you have the basic configuration in place, restart the service and validate — telnet to port 25 and send a test email.

There you have it, amazing Hyper-V virtualization — same capabilities as VMware vSphere with 25 percent fewer layers!

Related posts:

1. Choose any two: Hyper-V, HA, Linux
2. Even GPL can’t make Hyper-V Linux VMs well
3. Which guest operating systems can be customized by SCVMM?
4. Hyper-V Linux integration components no longer Connected

The Cost Per Application Calculator makes it clear that investing in VMware vSphere 4 significantly reduces your datacenter hardware footprint and associated costs.  Scott Drummonds, the VMware performance expert, recently explained how memory overcommit is the only way to effectively use all of the physical RAM in a hypervisor.

Each time this topic comes up, Microsoft revs up their marketing machine and responds like this:

• Hyper-V is free, use your money to buy RAM upgrades instead of VMware vSphere licenses
• Memory overcommit is risky, VMware tells customers to turn it off anyway

The truth is that Hyper-V will have memory overcommit the moment Microsoft figures it out.  If that day ever does come, watch the messaging quickly change to the familiar, “our customers asked us to implement this…” line.

Why is it fair to make such a bold claim?  Two words:

Dynamic Memory

It’s a little known fact that some of the early releases of Hyper-V R2 actually had a feature that allowed administrators to assign more RAM to virtual machines than was physically present on the host — “Dynamic Memory” a.k.a. memory overcommit.

We can look back and see what bloggers like Mark Wilson had to say:

Microsoft also spoke to me about a dynamic memory capability (just like the balloon model that competitors offer). I asked why the company had been so vocal in downplaying competitive implementations of this technology yet was now implementing something similar and Ward Ralston explained to me that this is not the right solution for everyone but may help to handle memory usage spikes in a VDI environment. Since then, I’ve been advised that dynamic memory will not be in the beta release of Windows Server 2008 R2 and Microsoft is evaluating options for inclusion (or otherwise) at release candidate stage.

And this blog provided more detail on the implementation:

In Hyper-V 1.0, physical memory was hard allocated to the VMs, but in 2.0 the pool of memory is dynamically allocated and removed based VM usage with no service interruption. Dynamically allocating memory to VMs can drastically improve host consolidation rates.

Hyper-V 2.0 VMs are configured with an initial RAM setting (how much the machine boots with) as well as minimum and maximum RAM values. Hyper-V then adds RAM using the Hot-Add function, and removes it using a balloon driver (for supported OSes).

For the visual learners in the crowd, take a look at this VM configuration dialog:

Note that this is nothing like Transparent Page Sharing in VMware ESX — Hyper-V VMs would actually be reconfigured using hot-add memory, so the guest operating systems must cooperate.

In an interview with Steven Bink, even Bob Muglia acknowledged the need for overcommit:

We talked about Vmware ESX and its features like shared memory between VMs, “we definitely need to put that in our product” later [Muglia] said it will be in the next release. Like hot add memory, disk and nic’s will be and Live migration of course, which didn’t make it in this release.

By the way, hot-add memory didn’t make it into Hyper-V R2, either — VMware ESX 4 has it today.

Quick Memory Overcommit?

Evidently, dynamic memory was not even up to the rigorous “Quick” standard and was dropped from the release train.  Perhaps in some future edition of Hyper-V, Quick Memory Overcommit will be offered — with just a few seconds of VM downtime as RAM allocation is dynamically adjusted.  But that’s  just speculation.

Sour Grapes

Instead of finding a way to implement memory overcommit in Hyper-V R2, Microsoft has taken the alternate approach of attacking VMware and declaring the feature unnecessary, unsafe, or too expensive.

The fact is that memory overcommit is an extremely valuable capability and VMware ESX has had it all along.  Some of the Linux-based hypervisors are starting to figure it out.  Until Hyper-V finally adds the feature we’ll continue to hear how easy it is to simply buy more RAM.

How long can you afford to wait?

Related posts:

1. Hello Dynamic Memory?
2. Exclusive: Hyper-V 2013 Product Roadmap
3. The Truth About Storage Hot Add and Remove
4. Hyper-V More Stable, Just as Mature as VMware vSphere?

From What’s New in VMware vSphere 4.0:

Enhanced ESX Error Reporting — vSphere 4.0 gives you the ability to report per-host network, storage and VMFS issues to the vSphere Client in the Events tab. Each error event also provides a link to a specific Knowledge Base article that explains the event and provides detailed remediation steps to resolve the error. vSphere 4.0 also has the ability to set specific alarms for such events, with the help of the Events and Alarms feature. See vSphere Basic System Administration.

I experienced the feature firsthand recently while reconfiguring some physical switches in my lab, causing the following event:

Clicking the link took me to KB 1009542 with much more specific detail about the warning.

Very nice enhancement.

Related posts:

1. PowerShell Prevents Datastore Emergencies
2. Easily view guest IP addresses with vSphere Client
3. Upgrading VMs to vSphere virtual hardware
4. VMware vCenter Server 4 task and event retention

VMware VI and vSphere SDK: Managing the VMware Infrastructure and vSphere by Steve Jin is a new book that offers something for a variety of VMware techies.

Steve Jin is the creator of the VI Java API, an open-source alternative to the official VMware SDK that overcomes some of the challenges of the native Web Services interface.  The primary advantage of the VI Java API is improved productivity due to a serious reduction in code — up to 70 percent.  Plus, it gives developers object-oriented programming and compile-time type checking not offered in Web Services.  But don’t mistake this API for an performance-degrading wrapper — it actually performs better than the two official VMware Java SDKs!

Even for non-developer types, it is quite easy to kick the tires — just download the Eclipse IDE for Java, the VI Java API, some sample code like PrintInventory.java, and fire it up.  This tutorial is helpful to get the project set up correctly.

As for the VI SDK, system administrators do not need to even read a line of code to benefit from this book.  Taking time to understand the ways in which various components interact and function will improve the overall understanding of a VMware vSphere environment — not to mention troubleshooting methodologies.

On the other hand, those managing VMware vSphere with the PowerCLI may encounter limitations from time to time that necessitate interacting with the SDK instead of the excellent PowerShell cmdlets.  Doing so is not for the faint of heart — LucD recently elucidated this process.  Therefore, even if you never intend to write a single line of VI Java API code, gaining a better appreciation of the VI SDK may just help you with your PowerCLI scripts.

If you have made VMware part of your career, it makes sense to consider this one for your bookshelf.  Also check out Duncan Epping’s review.

Disclaimer: The publisher provided a review copy of the book, but did not influence the content of this article.

Related posts:

1. Even GPL can’t make Hyper-V Linux VMs well
2. VI Toolkit (for Windows) 1.5 released
3. What’s New (and What’s Vapor) in SCVMM 2008
4. VMware vSphere 4 has a Snapshot Alarm

If you are not familiar with CapacityIQ (a.k.a. CapIQ or CIQ), it is an add-on management product for vCenter Server that assists in planning and managing virtualization resources.

CapIQ is very easy to deploy since it ships as a virtual appliance in OVF format — just load it onto an ESX host and perform a few simple configuration steps. It’s also simple to use since the user interface is a vSphere Client plug-in.  That means no additional web interfaces or user accounts to deal with — just a single pane of glass.

To learn more, check out the Evaluator’s Guide and other documentation.

Installation Experience

For those curious about the installation experience, here is an overview of the very simple procedure.

Upload the CapacityIQ VM using vSphere Client

Configure Credentials

The CapacityIQ appliance has two user accounts: root and ciqadmin.  When you first power on the VM, you will be prompted to enter passwords for those accounts.  Unlike some VM appliances, there are no default passwords involved.

Connect CapacityIQ to vCenter Server

After the appliance is finished booting, a URL will be shown on the console.  Open that with your browser and log in using the ciqadmin account.

Enter the address and credentials needed to connect to your vCenter Server:

vSphere Client Interface

After CapacityIQ is registered with your vCenter Server, restart your vSphere Client.  Verify that the plug-in has been installed automatically.

If everything is in order, you’ll find a new icon on the vSphere Client Home screen:

Views and Reports

CapacityIQ comes with a number of preconfigured views and reports, as seen here:

Related posts:

1. VMware vCenter Server 4.0 installation video
2. Installing vCenter on Linux Technical Preview
3. VMware vCenter Server 4 task and event retention
4. VMware vCenter Server performance stats levels

• vStorage VMFS Volume Grow
• Hot Extend for Virtual Disks
• Hot Virtual Disk Add/Remove

With these capabilities, if space gets tight in your vSphere environment, it is easy to be proactive and address the issue before anyone notices.  The process goes something like this:

• Allocate additional physical hard disk space on your SAN to an appropriate LUN
• Grow your VMFS datastore onto the newly added free space
• Extend, or add new, virtual disks for the VMs that need more storage
• Expand the volume inside the guest to create more usable space

All without a reboot: zero downtime.

Not All Virtualization Platforms are Created Equal

Don’t jump to the conclusion that all hypervisors offer the same flexibility.  Perhaps you are wondering about Hyper-V capabilities?

First, let’s take a look at some Microsoft System Center Virtual Machine Manager 2008 R2 (SCVMM) marketing statements:

The What’s New page announces:

Hot addition/removal of Storage: Allows the addition and removal of storage to virtualized infrastructure without interruption. Additionally, “live” management of virtual hard disk (VHDs) or iSCSI pass through disks, allows administrators to take advantage of additional backup scenarios and readily use mission critical and storage-intensive applications.

The Top Benefits list proclaims:

Hot addition/removal of storage: With this capability, administrators can quickly and efficiently respond to changing storage requirements of virtual machines. This ability to hot-add additional storage eliminates the previous need to take the host down to upgrade storage thus increasing business continuity for end users and reducing complexity for administrators. Additionally it allows administrators to confidently deploy mission critical applications (in which up-time is of paramount importance) that may have rapidly changing storage requirements such as web, database or other business applications.

An IT decision-maker just might get the impression that both ESX and Hyper-V have essentially the same features. They do not.

You may be surprised to find out that all of the descriptions above merely refer to adding a new virtual disk to a VM — providing the conditions are right.  You cannot grow an existing VHD, and you can’t safely remove a VHD with SCVMM.

Plan Ahead

First things first.  If your Hyper-V VM does not have a virtual SCSI adapter — templates and VMs from Hyper-V R1 do not — you won’t be able to hot add a new VHD until you correct that shortcoming.  Hello downtime.

SCVMM can add a new blank virtual disk to a VM or it can copy an existing one across the network from the Library — if you copied it there beforehand, but there is no way to add a VHD that may already be present on your SAN — even if it is already sitting right next to the destination VM.

Removal?

Up-time may be of paramount importance, but preventing data-loss was evidently not part of the original design.  Removing a VHD with SCVMM results in the immediate deletion of the underlying VHD file.  Ouch!  Thankfully, a recent patch improves administrator job security by throwing up a warning before this happens, providing an option to cancel.  There is no way to simply disconnect a VHD using SCVMM.

It turns out that if you really want to take advantage of those “Additional Backup Scenarios” by hot adding and removing virtual disks, you need your trusty Hyper-V Manager utility.  Still think System Center is a single pane of glass?  And that’s not the only task that requires administrators to switch between Hyper-V interfaces.

Conclusion

Microsoft is trying hard to ride the coattails of VMware ESX.  The latest release of Hyper-V R2 still does not have capabilities enjoyed by VMware administrators since the ESX 3.5 days.

Don’t believe the obfuscated marketing literature.  VMware vSphere is for real.

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Responsible Thin Provisioning in VMware vSphere
3. The Truth About VM SAN Transfers
4. Easy recovery from a full VMware ESX datastore

Under the supervision of VMware vCenter Server, ESX 4 thin provisioning is safe and reliable even for production workloads thanks to advanced storage accounting and built-in monitoring.  And even if the worst does happen — an unexpectedly full datastore — recovery is simple.

Now that you are convinced that VMware vSphere thin provisioning has a place in your data center, you may be wondering about performance tradeoffs.

It turns out that thin disks perform just about as well as thick disks.

New Performance Results

In this brand-new VMware vStorage Thin Provisioning performance study, VMware performance engineers compare thick and thin disk performance.  This must-read document covers several important topics, such as:

• I/O-intensive benchmarking with a 16-node ESX cluster
• File copy benchmarking with 2 ESX hosts
• Fragmentation impact
• Thin provisioning affect on co-located thick disks

Now, go forth and provision… thinly.

Related posts:

1. Finding thin-provisioned virtual disks with PowerShell
2. Responsible Thin Provisioning in VMware vSphere
3. Easy recovery from a full VMware ESX datastore
4. The Truth About Storage Hot Add and Remove

• Full accounting for all provisioned storage space and complete monitoring
• Automatic response to a nearly-full datastore through vCenter Alarms, PowerShell, and Storage VMotion

You may be nearly convinced to start using thin provisioning, but still wondering…

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically grow to accommodate additional storage demand.  When VMware ESX detects this condition, virtual machines in need of additional storage are instantly paused to prevent guest operating systems from failing. Conversely, VMs that that read and write to existing allocated storage blocks will continue running without issue — not all virtual machines will be paused just because a datastore is out of space.

If you ever find yourself in this situation, it’s not hard to fix.  Here is one simple approach, step-by-step:

1. Free up some space by deleting or moving files — ISO images or powered-off VMs would be perfect
2. Resume one of the paused VMs
3. Use Storage VMotion to move the disks for that VM to another datastore
4. Resume the remaining VMs

Watch the procedure in action:

Depending on the size and storage demand of each VM, additional migrations may be needed.  An alternative resolution would be to add additional space to the SAN LUN and grow the VMFS volume.

The Experiment

To simulate a sudden storage demand by the thin-provisioned VMs in the above video, I simply copied a large file from a network share to each Windows Server 2003 VM simultaneously.

For the curious, below is a PowerShell script for the task.  Run it from anywhere — it uses Sysinternals psexec to remotely initiate a file copy on each VM from a network share.

# VM naming convention combines this string with 2-digit number
$vmPrefix = "VM2003e-"
 
# UNC path to a large file that will be copied into each VM
$iso="\\fileserver\ISO\large.iso"
 
# credentials to download above file from \\fileserver
$user="domain\username"
$pass="pa55word"
 
# pass this function a list of numbers
function createVmList ($series) {
    $vmList = @()
 
    foreach ($id in $series) {
        $vmList +=  %{"$vmPrefix{0:00}" -f $id}
    }
    $vmList
}
 
# make sure Sysinternals psexec is in your path
function psexecOnVm ($cmd, $vmList) {
    $vmList | % {Invoke-Expression "psexec.exe \\$_ -d -u $user -p $pass $cmd"}
}
 
psexecOnVM "cmd /c copy /Y $iso c:\" (createVMlist (31..39))

VMware ESX is Resilient

You may have been surprised at how easy it is to recover from a full datastore — without so much as a guest OS reboot.  It’s a testament to the rock-solid architecture behind VMware ESX and VMFS.  No other virtualization platform comes close.  Try for yourself.  See what happens if a group of thin-provisioned Hyper-V virtual machines suddenly run out of storage — it’s not going to be pretty.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. vSphere Thin-Provisioned Disk Performance
3. PowerShell Prevents Datastore Emergencies
4. Finding thin-provisioned virtual disks with PowerShell

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Related posts:

1. Easy recovery from a full VMware ESX datastore
2. Responsible Thin Provisioning in VMware vSphere
3. VI Toolkit (for Windows) 1.5 and the PowerShell prompt
4. VMware vSphere 4 has a Snapshot Alarm