What happens when the other major component of a RHEV infrastructure fails?  Can you rely on RHEV High Availability (HA) to quickly and reliably restart affected VMs when a RHEV Hypervisor fails? It depends – as you will see.

First, let’s make sure everyone is up to speed on HA capabilities provided by the gold standard in virtualization:

VMware HA

VMware HA is a robust feature that was first introduced with Virtual Infrastructure 3 in 2006.

VMware vCenter Server is required to configure HA options and add VMware ESX hosts to a cluster, but after that vCenter is hands-off — ESX hosts communicate among themselves to reliably restart virtual machines.  In fact, VMware HA can even restart vCenter Server if it is running inside a protected VM — wrap your head around that one.

Powerful options are available for administrators, such as specifying the restart priority of virtual machines and whether or not to force VMs to power off if a host becomes isolated from the rest of the cluster.

VMware has heavily invested in this technology, reducing risk  for customers that virtualize with vSphere.  For even more information on VMware HA, take a look at Duncan Epping’s HA Deep Dive.

RHEV HA [ha ha]

Looking at this Red Hat Enterprise Virtualization competitive comparison, you’d might assume that RHEV and vSphere are on equal footing when it comes to protecting virtual machines with HA:

Unsightly details behind the marketing

RHEV HA sounds great in the marketing brochure, but there are a few problems with the execution.  RHEV Manager is a single point of failure — running on a physical Windows box — and it’s also the actual brain behind HA.  Yes, RHEV-M is responsible for restarting virtual machines when a host fails.  If the manager is down, no HA for you!

That alone makes RHEV HA something less than “HA” for most production environments, but there are a few other key weaknesses:

• HA must be manually enabled for each virtual machine — no cluster-wide settings
• No cluster admission control — administrators must manually ensure sufficient capacity would be available in a cluster to accommodate a host failure
• No VM restart priority to ensure the most critical workloads and dependencies are brought online first
• Primitive split-brain protection requires IPMI or other out-of-band management interface to force a host shutdown
• Cannot protect the RHEV Manager itself — chicken-and-egg situation

Wow, I didn’t notice those details in the comparison brochure.

Decide

Whether your datacenter is running Windows Server or the mighty Red Hat Enterprise Linux, doesn’t it makes sense to trust the proven leader in virtualization?  VMware vSphere is simply the most reliable platform for consolidating workloads and building your private cloud.  Going beyond exceptional HA is VMware FT – mirroring mission-critical VMs on backup hosts means zero downtime from host failures.

Related posts:

1. Red Hat Enterprise Linux is not Enterprise Virtualization
2. What is Red Hat Enterprise Virtualization?
3. RHEV Manager — It’s not just a clever name
4. Red Hat Enterprise Virtualization: Pentium II Inside!

You have probably also heard over and over about Hyper-V and High Availability (HA) virtual machines.  And please, don’t forget — HA is free!

Should one conclude that a supported Linux VM on Hyper-V would be fully functional in an HA configuration?

No, one should not.

Hyper-V HA is based on Microsoft’s general purpose Failover Clustering — originally designed for applications like Exchange and SQL Server.  If you can say one thing about MSCS, it tries very hard to keep managed services online.  Sometimes too hard.

In fact, without tight integration between Clustering and the protected services, it’s especially difficult to stop things in an orderly fashion.  System Center Virtual Machine Manger 2008 has this integration, and stopping an HA VM with the SCVMM Administrator Console will normally initiate a proper shutdown.  But, in order to do that, guest operating systems need to have integration components that offer the “Operating system shutdown” service .

Linux integration components for Hyper-V do not offer this service, and the following error is seen when attempting to shut down a Linux VM via SCVMM:

The next thing to try is shutting down the Linux VM from the Failover Cluster Manager UI — one of the many other “single panes of glass” needed to manage your Microsoft virtualization infrastructure.

Evidently, this approach does not perform the same validation as the SCVMM interface — there is no error and the guest OS actually does not shut down cleanly — the VM is simply stopped.  It turns out that, like physical machines, virtual machines don’t really enjoy being abruptly powered off.  Here you can see this poor filesystem being repaired during the next boot:

It appears that the only option for shutting down a Linux VM on Hyper-V is through the guest interface itself.

Unfortunately, when the VM shuts down, Failover Clustering notices immediately and assumes there has been a failure.  One restart, coming up…

With Hyper-V you can run HA virtual machines and you can run Linux virtual machines, but you can’t run HA Linux virtual machines.

The only hypervisor that offers full functionality for both Windows and Linux guest operating systems is VMware ESX.

Related posts:

1. Even GPL can’t make Hyper-V Linux VMs well
2. IGT Part 4: Failover frustration
3. Hyper-V Linux Smokescreen
4. Hyper-V Linux integration components no longer Connected

Have you ever wondered what life in the virtualization world would be like without a clustered filesystem? Thanks to Microsoft Hyper-V, NTFS, and their failover cluster service, you no longer have to wonder, you can read about it in great detail.

In a world without VMFS, in order to have HA virtual machines or quick migration (let’s not bring up live migration today, since it is shipping “in 2010″) here is the most important point: Each virtual machine must be on its own LUN. Are you on good terms with your SAN administrator? Looks like the two of you are going to be chatting quite often.

Now if you prefer, you could set up several virtual machines on one LUN. But it turns out that such a plan would be destined for failure. No pun intended. Please take a look at the following quote from a Microsoft Technet blog:

If you group clustered VMs on a single LUN, don’t shut down the OS from within the OS or from Hyper-V Manager, this is not a cluster-aware shutdown and counts as a failure. The OS will be restarted by the cluster and then if you go “what the ?!?!” and shut it down again, depending on how the cluster resource is configured this will induce a failover to another node, taking the rest of your VMs with it. Try explaining that outage to your boss when a business critical server goes out of action for a couple of minutes at a crucial period because you shut down a scratch VM you had been using to test some software. I for one do not feel like having to get a change request authorized just to shut down a machine. Once you have made a VM a clustered VM, ALWAYS use Failover Cluster Management or the SCVMM Console to shut it down.

Ouch! What happens on Patch Tuesday? Let’s not find out.

One last thought: What about SAN deduplication technology that operates per-LUN?

Related posts:

1. IGT Part 4: Failover frustration
2. IGT Part 3: One-VM-per-LUN doubters
3. Choose any two: Hyper-V, HA, Linux
4. Hands off that CSV!