However, one limitation that some have discovered while using this lab-testing technique is the lack of ability to use snapshots with virtual ESX systems.  In fact, after taking a snapshot of a virtual ESX VM, you will see the system boot into the recovery shell like so:

Ouch.

Can we fix it?

I asked some super-smart engineers inside VMware about the issue, and of course they know all about it and how to make it work.

The solution is to enable an undocumented advanced configuration option on the physical ESX host like so:

In other words:

• Log into the console of the physical VMware ESX 4 host
• Run the following command: esxcfg-advcfg -s 1  /COW/PermitVmfsOnRedoHierarchy
• Verify the setting with the “get” flag: esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy

Do I even need to say that this stuff is not supported by VMware Global Support Services (GSS)?  Well, I just did.

Yes we can!

After executing that one command on your physical machine, take a snapshot and boot up your virtual ESX:

Everything works just like you need it to now — you can even create a snapshot hierarchy:

Now what?

Were you constrained by the lack of virtual ESX snapshots?  What interesting things will you use this for?

Related posts:

1. Half the VM reboots on Patch Tuesday
2. VMware vSphere 4 has a Snapshot Alarm
3. Hyper-V snapshots: not for production
4. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off

What’s new is not necessarily the technology — it’s the management.  In fact, veteran VMware ESX admins have been creating thin provisioned virtual disks for years — for controlled scenarios — by way of the vmkfstools command.

Before vSphere and ESX 4, however, thin disks came with risk — there was no simple way of accounting for the overcommitted storage on each LUN.  Even with multiple gigabytes of free space, a small gang of thin-provisioned virtual machines could quickly quickly grow to exceed datastore capacity during a sudden demand spike.

Complete Storage Accounting

Now in vSphere 4 there is a new element in the capacity section of the datastore summary tab that shows total provisioned space — the maximum potential growth of all virtual machines if thin provisioned disks were fully utilized:

Virtual machines with snapshots have the potential of consuming even more datastore space, so vSphere accounts for this condition, too.  Take a look at this VM Summary tab, where the total provisioned storage includes:

• Virtual hard disk (40GB)
• Snapshot (another 40GB potential, worst-case)
• VM swap file (1GB — sized according to RAM in theVM)

As you can see, a VM with a 40GB virtual disk can actually consume up to 81GB of space on your SAN because of a forgotten snapshot!  Use the  snapshot alarm to stay in control.  And don’t forget that VMware vSphere snapshots are perfectly suitable for production.

New Datastore Alarms

New alarms in vSphere prevent out-of-space surprises.  Administrators can monitor not only the free space on a datastore, but also the percentage overallocated — making it easy to adhere to policies concerning thin provisioning aggressiveness in your environment.

Flexible Virtual Disk Re-configuration

When creating a new VM, opting for thin-provisioned disks is as easy as checking a box.  If you change your mind later, you don’t have to start over — during a Storage VMotion operation, administrators can opt to change the VM disk format on the fly (see below).  It is also possible to inflate thin disks into thick via a new menu in the Datastore Browser.

Danger!

You’ve probably heard the now-famous quote by Tom Bittman from Gartner:

Virtualization without good management is more dangerous than not using virtualization in the first place.

That goes double for thin-provisioned virtual disks.  Without comprehensive accounting and monitoring in place, your virtual infrastructure may be heading for disaster.  This level of insight is only available with VMware vSphere 4 — and it’s built right into the platform.

What about Microsoft virtualization?  Hyper-V R2 thin provisioning — known as “dynamic disks” — is not a best practice.  Perhaps due to the lack of accounting and monitoring of storage overcommitment — especially critical now with Cluster Shared Volumes and multiple VMs per LUN.

Are you using vSphere thin provisioning?

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Easy recovery from a full VMware ESX datastore
3. Finding thin-provisioned virtual disks with PowerShell
4. VMware vSphere 4 has a Snapshot Alarm

If you happen to want a simple, built-in way to monitor your snapshots in VMware vSphere 4, look no further than this new alarm:

Very nice addition.

Related posts:

1. Responsible Thin Provisioning in VMware vSphere
2. Half the VM reboots on Patch Tuesday
3. Taking snapshots of VMware ESX 4 running in a VM
4. Hyper-V snapshots: not for production

Previously, I have written about some of the shortcomings with Microsoft’s Hyper-V snapshots.  Or were they checkpoints?  I forget… but that’s not important.  The main problem with the Hyper-V snapshot implementation — that does not affect VMware ESX — is that the snapshots are not actually removed until the VM is powered off.  All the way off — reboots don’t count.  And the VM must remain powered off until the AVHD file is completely merged, otherwise the process is aborted.

By the way, if you are using System Center Virtual Machine Manager 2008, you will have no way of knowing when the snapshot merge is complete.  SCVMM just simply does not report this information — switch to the other single pane of glass (Hyper-V Manager) if you need to be sure.  Hyper-V administrators are used to this by now.

So, if you find your VM in “Paused-Critical” state, you might want to check for a hidden snapshot file filling your LUN.  If you’ve landed on VCritical in search of such a solution, you are not alone — quite a few Google searchers end up here due to this issue.  Before Ben Armstrong’s recent — and very informative — series on snapshots, VCritical was among the first to elucidate this issue.

If a hypervisor’s snapshot functionality is not recommended for production, is that hypervisor itself ready for production?

Related posts:

1. Half the VM reboots on Patch Tuesday
2. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off
3. Snapshots that shoot back
4. VMware vSphere 4 has a Snapshot Alarm

Evidently, some Hyper-V administrators are puzzled by that platform’s snapshot behavior.  I am starting to see visitors come to VCritical when searching for:

hyper-v “merge in progress”
hyper-v “paused-critical”

This is exactly the issue that I wrote about recently.  With VMware ESX, administrators can create and delete snapshots on running VMs — without incurring downtime.  Deleted Hyper-V snapshots are only merged when the VM is powered off.  That could mean twice the number of reboots during VM guest OS maintenence.

Related posts:

1. Hyper-V snapshots: not for production
2. Snapshots that shoot back
3. Half the VM reboots on Patch Tuesday
4. Exclusive: Hyper-V 2013 Product Roadmap

Even if you are not using VMware Update Manager, a typical patch workflow may look something like this:

1. Take VM snapshot
2. Apply updates, which normally require that you…
3. Reboot
4. Test
5. Remove snapshot

After I wrote yesterday’s post on Hyper-V snapshots, something occurred to me.  The 5 steps above are only for VMware ESX.  If you are using Hyper-V as your platform, you are not done yet — the snapshot differences are not actually merged until the VM is powered off.

Hyper-V administrators, please add the following steps:

6. Power off VM
7. Wait for merge to finish
8. Power on VM

Patch Tuesday. Would you like one reboot or two?

Related posts:

1. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off
2. Hyper-V snapshots: not for production
3. Snapshots that shoot back
4. VMware vSphere 4 has a Snapshot Alarm

Snapshots are one of the great features of virtualization, but forget about one on an active VM for too long and you may be calling for cleanup on aisle one.  Recently there was a fair amount of discussion among VM bloggers about snapshots.  Jason Boche had a nice article that summarized some monitoring strategies and Rich Brambley also weighed in.

You may be wondering what makes snapshots so great.  At the top of the list is the ability to have an easy roll-back plan for failed upgrades.  For example, an administrator can take a VM snapshot before applying patches to a guest OS.  After rebooting and testing, if all is well, simply delete the snapshot and move on — the changes are merged while the virtual machine is running.  If for some reason a patch caused problems, the previous state can be quickly restored.

That’s how VMware ESX works.  Is Microsoft Hyper-V the same?

It turns out that Hyper-V snapshots (or SCVMM checkpoints, depending on which single pane of glass you are using) use a slightly different design. The Microsoft interfaces allow administrators to remove snapshots from powered-on VMs, and after doing so, they appear to be gone:

But, upon closer inspection, the changes are not actually merged until the VM is powered off:

What happens if an administrator is not aware of this behavior?  Just as if the snapshot was never deleted, it grow and grow until the LUN on which it lives runs out of space:

And when that happens, the VM will go into a Paused-Critical state:

Cleanup on aisle one!

What is the best way to recover gracefully from such a problem, if there really is no free space on the LUN? Gabes Virtual World published a great tip that could help next time.  Maybe all that ISO file copying isn’t so bad after all.

Related posts:

1. Hyper-V snapshots: not for production
2. Half the VM reboots on Patch Tuesday
3. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off
4. Checkshots or Snappoints?

What’s the difference between a checkpoint and a snapshot?

Two Microsoft development groups.

Don’t get it?  Let me explain…

Hypervisors offer a means of saving the current state of a virtual machine and later returning to the saved state, if desired.  This is extremely useful when performing repetitive testing procedures, and also as a contingency plan in case an application upgrade fails.

VMware products use the term “snapshot” for this technology, and so does Hyper-V.  Surprisingly, even though SCVMM manages Hyper-V and ESX, it uses a different term for the exact same feature.  SCVMM calls them “checkpoints” and evidently this is confusing enough for Microsoft users that they have even called Microsoft support to ask about the differences.

Microsoft has two completely separate interfaces, with different capabilities and terminology, for just their hypervisor.  Can you seriously believe the rhetoric about Microsoft’s single pane of glass to manage your entire environment?

Administrators use the same exact VI Client whether they are connecting to an individual VMware ESX host or centrally managing many hosts with VirtualCenter.  No need for multiple terminology or interfaces.

Related posts:

1. Hyper-V snapshots: not for production
2. Half the VM reboots on Patch Tuesday
3. Snapshots that shoot back
4. Single Pane of Glass — Hyper-V Edition