The key claim — that Microsoft offers more control over virtual machine memory — is misinformed at best:

Hyper-V Dynamic Memory also has a greater range of configurable options than does VMware memory overcommit. Users can assign limits to problematic VMs with memory-hungry workloads, and if memory contention occurs, users can prioritize specific VMs. A configurable buffer value also identifies how much extra memory is reserved for short-term needs between rebalancing passes.

Readers seeking to find the true differences between these platforms will need to search elsewhere.  A factual comparison reveals that Hyper-V Dynamic Memory offers no advantage over VMware vSphere:  A VMware ESX VM has settings for memory size, limit, reservation, and shares to specify priority.  Not only that, VMware vSphere offers a comprehensive range of memory management technologies: ballooning, page sharing, compression, and host swapping.

What vSphere doesn’t have is a reserve memory buffer setting — this is an artifact of the Dynamic Memory design, accommodating for lag time inherent to the hot-add process.  Interestingly, when running important enterprise applications like SQL Server, Microsoft recommends cranking the buffer down to the lowest possible setting.

VMware vSphere offers the widest range of memory management and configuration capabilities, accommodating even the most demanding workloads.

Related posts:

1. Hyper-V [Not Exactly] Dynamic Memory
2. The Truth About Hyper-V Memory Overcommit
3. VMware vSphere 5 outperforms Hyper-V by nearly 20%
4. Hyper-V Dynamic Memory: Not Quite Ready to Demo!

Two different vSwitches are provided in vSphere: Standard and Distributed.  Both offer NIC teaming for load balancing and fault tolerance, intuitive VLAN support, and Cisco Discovery Protocol (CDP) for easy lights out datacenter management.  Be sure to check out Jason Boche’s great overview of CDP.

vSphere Distributed Switch — Simple Network Management

The Distributed Switch adds advanced networking features to your virtual infrastructure, such as load-based teaming and private VLANs, and offers centralized port group management — eliminating the need to configure vSwitches and port groups individually on each host.  vSphere administrators can also choose to go with a hybrid model, maintaining a Standard vSwitch on each host — typically for management — and leveraging a Distributed Switch for virtual machine traffic.

Here you can see a Distributed Switch that spans four ESX hosts, utilizing two physical NICs per host :

By connecting these physical NICs to multiple trunk ports, virtual machines benefit from network redundancy and load balancing while making it trivial to create port groups for any VLAN required.  Configurations are instantly propagated across the cluster, boosting efficiency and minimizing human error.

Also shown above is the CDP information for one of the physical NICs, handy for identifying connected physical switch ports — no need to convince a coworker to visit the datacenter and help trace cables!

Hyper-V NIC Teaming Update

It’s been a while since I last wrote about the state of Hyper-V NIC teaming, some readers may be curious to know if anything has changed.  Let’s start our investigation by reviewing the latest official Microsoft support policy:

Since Network Adapter Teaming is only provided by Hardware Vendors, Microsoft does not provide any support for this technology…

Since Microsoft Hyper-V Virtualization is a new technology, we recommend that you thoroughly test your teaming solution in a test environment prior to deploying into Production.

Looks like it’s up to the end user to put this puzzle together.  Fortunately, enthusiastic bloggers have stepped up to provide some of the missing guidance to help decide which combination has the best chance of success.  Hyper-V expert Hans Vredevoort begrudgingly admits:

Now that Windows Server 2008 R2 SP1 was published almost a month ago, I have attempted to evaluate the consequences for this lovely combination.

There are just a few main elements to consider before attempting to bring robust private cloud networking capabilities to the Windows hypervisor:

• Windows Server service pack and hotfixes
• Third-party network drivers
• Third-party network teaming drivers
• VLAN requirements, and at what level in the stack to implement

[Math quiz: calculate the number of possible combinations administrators must consider.]

What’s all the fuss?  Just download the latest versions of everything and go.  Not so fast…

Experts Offered No Immunity

Stu Fox, Microsoft virtualization expert and VCritical reader, recently shared his experience with this very issue — encountering an outage after a simple driver update:

Hope for the Future?

It’s hard to say if Microsoft will ever integrate NIC teaming capabilities directly into the Windows platform; the next version is expected sometime next year.  For now Hyper-V users will just have to rely on the upcoming Virtual Machine Manager 2012 release, which will no doubt introduce complexity abstraction to VM networking in an effort to bring it closer to the simplicity vSphere administrators have enjoyed for years.

Is it any wonder Gartner concludes that Hyper-V is under-performing?

Related posts:

1. Yes, NIC teaming is not unsupported
2. The Secret of Ephemeral Port Groups
3. Thanks for all the port groups!
4. Single Pane of Glass — Hyper-V Edition

In a homogeneous VMware ESX cluster, all CPU instructions in the underlying host CPU are exposed to guest operating systems.  VMware vCenter Server also offers the state-of-the-art Enhanced VMotion Compatibility (EVC), allowing administrators to specify a baseline in a mixed cluster that maximizes use of most modern CPU features during transition to newer generation hardware.

VMware ESX 4 CPU Features

CPU-Z is a handy utility that displays comprehensive technical information about a CPU.  Here it is running on a VMware ESX VM powered by an Intel Xeon E5540 “Gainestown” processor — a Nehalem-class CPU with most of  the latest instructions, such as SSE4.1 and SSE4.2:

Virtual machines running on VMware ESX are able to take advantage of those features that are designed to improve application performance — giving you your money’s worth after investing in new hardware.

Now let’s take a look at an alternative approach…

RHEV Hypervisor: Pentium II Inside!

You probably know by now that Red Hat has abandoned Xen in favor of the more modern KVM.  So how did the architects behind Red Hat Enterprise Virtualization (RHEV) deal with CPU compatibility across KVM hosts?

They decided not to expose the true underlying CPU details and advanced instruction sets.  Instead, they take a more conservative approach and masquerade as an old Pentium II Celeron CPU — no pesky SSE4 instructions to deal with here:

That’s the virtual CPU presented to guests even in a cluster of identical Nehalem hosts!  This is as “good as it gets” for RHEV.

Now, to be fair to KVM — the open source project on which RHEV is based — the capability to expose all physical CPU features is present, it’s just that Red Hat has not designed a suitable management feature such as VMware EVC to take advantage of it.

Red Hat claims that RHEV is just like vSphere, but cheaper (where have we heard that before?).  Is it?

The Hypervisor is not a Commodity

Are you buying new hardware for your virtualization project, paying a premium for hardware with the latest performance capabilities?  Keep in mind that not all hypervisors are created equally.  Only VMware has EVC — and that’s just one of the many reasons why vSphere is the best platform for your Red Hat Enterprise Linux and Microsoft Windows Server workloads.  Why build virtual silos in your datacenter with OS-centric hypervisors rife with compromises?

Get the best, most mature virtualization: VMware vSphere.

Related posts:

1. What is Red Hat Enterprise Virtualization?
2. Red Hat Enterprise Linux is not Enterprise Virtualization
3. Red Hat Enterprise Virtualization HA [ha ha]
4. Dear Red Hat…

I know it’s been a while — I’ve just needed some time to process what happened between us.

It’s hard to believe that we’d end up here, after all we’ve been through together.  When I needed a service console to manage my hypervisor, you were my first choice — and you were always there for me.  Well, except for those times when I badly needed some upstream RPMs updated… but I’m over that now.  Just about.

But over the years you changed and you are just not the distro I once knew.  I tried to look the other way when you started running around with Xen.  I knew it wouldn’t last.  Talk to Xen much anymore?  I didn’t think so.

How could you drag licensing into the picture?  For all the talk of open source this and GPL2 that, it never made sense that you would let anyone run unlimited RHEL guests only when you are a Xen Dom0 — but not when I am the hypervisor.  Isn’t a guest OS a guest OS?  I guess not.  You got selfish and frankly, people thought you were being a bully.

So you got tired of Xen and started looking around for satisfaction from another, huh?  How’s that KVM working out for you?  I hate to be the one to point this out, but your virtualization manager runs exclusively on Windows and is only accessible through Internet Explorer.  Have you completely forgotten that you are a Linux distribution! I hate for you to hear this from me, but Ubuntu and SUSE have been making fun of you for months.

You and your KVM are going to be too Linux-y for the Windows guys, and too Windows-y for the open source crowd.  I’m concerned about your reckless behavior.  That makes Red Hat Enterprise Virtualization the worst of both worlds.  You’re not thinking about hurting yourself, are you “RHEV”?  I would not even be surprised to find out that you use Bing as your search engine now.  Sorry, that was a low blow.

So you heard about the ESXi no-more-service-console thing, huh?  Well, I moved on, lost some weight, and am ready for the next phase now.

If you ever need a hypervisor to run on, you know where to find me.

Best friends forever,

VMware “Elastic Sky” ESX

[Disclaimer for the humor impaired: Eric Gray does not speak for VMware, nor does he speak for any personified software products.]

Related posts:

1. Red Hat Enterprise Linux is not Enterprise Virtualization
2. Red Hat Enterprise Virtualization: Pentium II Inside!
3. What is Red Hat Enterprise Virtualization?
4. Even GPL can’t make Hyper-V Linux VMs well

Windows deserves applause for reliability improvements in recent years.  Unfortunately, the most reliable Windows design will never be able to counteract the damage that can be inflicted by a misbehaving device driver.  In fact, take a look at this slide from a Mark Russinovich TechEd session where he makes the point that the majority of Windows blue screens (BSODs) are caused by third-party drivers:

Experts agree — Windows reliability is a function of driver reliability.

What about those drivers included with Windows?

You might be under the impression that these third-party drivers are for off-brand NICs purchased from the clearance bin at Fry’s.  That’s not exactly the case — plenty of drivers are included right on the Windows DVD.

Consider this recent situation detailed by Mark Wilson.  He set up a Hyper-V test machine with some Intel PRO/100 NIC cards and when he plugged in an Ethernet cable — instant BSOD!

Mark’s conclusion:

It’s good to know that Hyper-V was not at fault here: sure, it shows that a rogue device driver can bring down a Windows system but that’s hardly breaking news…

Meanwhile, back at the datacenter

Microsoft Hyper-V poster child Crutchfield recently found that the Broadcom NICs embedded in every one of their Dell servers were randomly losing virtual switch bindings after upgrading to Hyper-V R2.  Hello downtime!

The administrator left us with the following thoughts:

Currently, Microsoft doesn’t have a work around for this issue and nor is there an ETA for resolution. After all, it may not be Microsoft, but could very well be Broadcom.

So, what do we do? Well, in my configurations I can’t afford these little gotchas and I will be working only with my trusted Intel NIC’s for my virtual networks.

Uh, you might want to check with Mark Wilson first about Intel drivers and Hyper-V.

The chain breaks when the weakest link fails

Notice the consistent theme in these two scenarios?  The administrators are not blaming Hyper-V, instead focusing on the third-party hardware and drivers.  Folks, it just does not matter — a hypervisor failure is incredibly disruptive.

When selecting the foundation for your private cloud, will you choose a hypervisor based on a general-purpose OS that aims to support hundreds of hardware variations from across the spectrum?  Or will you build on a rock-sold, purpose-built platform that is specifically designed to aggregate and pool resources in your datacenter?

Related posts:

1. Yes, NIC teaming is not unsupported
2. The Million Hypervisor March
3. Hyper-V Linux integration components no longer Connected
4. Of Mice and Xen

“Yo dawg, I heard you like Patch Tuesday, so I put Windows in your hypervisor so you can patch it while you virtualize!”

Inspired by a joke from Kendrick Coleman. With apologies to Xzibit.

Related posts:

1. Let them eat cake!
2. Half the VM reboots on Patch Tuesday
3. A very flashy hypervisor: Hyper-V Server R2
4. Hyper-V Server Hyperbole

However, one limitation that some have discovered while using this lab-testing technique is the lack of ability to use snapshots with virtual ESX systems.  In fact, after taking a snapshot of a virtual ESX VM, you will see the system boot into the recovery shell like so:

Ouch.

Can we fix it?

I asked some super-smart engineers inside VMware about the issue, and of course they know all about it and how to make it work.

The solution is to enable an undocumented advanced configuration option on the physical ESX host like so:

In other words:

• Log into the console of the physical VMware ESX 4 host
• Run the following command: esxcfg-advcfg -s 1  /COW/PermitVmfsOnRedoHierarchy
• Verify the setting with the “get” flag: esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy

Do I even need to say that this stuff is not supported by VMware Global Support Services (GSS)?  Well, I just did.

Yes we can!

After executing that one command on your physical machine, take a snapshot and boot up your virtual ESX:

Everything works just like you need it to now — you can even create a snapshot hierarchy:

Now what?

Were you constrained by the lack of virtual ESX snapshots?  What interesting things will you use this for?

Related posts:

1. Half the VM reboots on Patch Tuesday
2. VMware vSphere 4 has a Snapshot Alarm
3. Hyper-V snapshots: not for production
4. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off

One outstanding achievement is the way the Hyper-V architects managed to completely collapse the guest operating system layer directly into the hypervisor.  Take a look at this diagram from a Microsoft Virtualization competitive brochure:

[Yes, it's that same brochure again.]

There are several key differences between Microsoft and VMware hypervisor architectures.  While the hardware and applications layers are pretty straightforward, applications don’t run directly on hardware.  As you are about to see, these two platforms take very different approaches when it comes to virtualizing workloads.

The VMware vSphere solution is to use a thin, high-performance virtualization layer that enables a single physical system to simultaneously host multiple virtual machines.  Each of these virtual machines executes isolated instances of traditional guest operating systems — Windows, Linux, Solaris, Netware, etc.

Due to apparent technological advancements in Hyper-V, it is now possible to run applications of all kinds directly on that integrated Windows OS/hypervisor layer without the need for a guest operating system at all.  This allows Microsoft to omit an entire layer from their virtualization architecture.

I’m sure you have seen various announcements about the exceptional support that Hyper-V has for Linux workloads.  Without a guest operating system layer, one must logically conclude that Linux applications are actually running on the Windows/Hyper-V hypervisor integration directly.  Spectacular!

I thought I’d test this technological breakthrough in the lab myself by running Postfix, a very popular Linux SMTP mail server, on Windows Server 2008 R2 with Hyper-V.  According to the diagram above, Postfix is classified as an application — the third layer.

Now, in order to install Postfix, I have to interact with the second layer in the stack, which is the “Microsoft Operating System w/Virtualization.”  Since I’m an RHCE, I decided to use a Red Hat Enterprise Linux package of Postfix, but we can all agree that SUSE Linux Enterprise Server 11 would work equally well.  However, I would not recommend using Ubuntu packages since that distribution is not supported by Hyper-V.  Another side benefit to this technique is the fact that Windows has great mouse support.

Just as one would do on a legacy Linux system, I opened a trusty command shell and installed the package like so:

This is the first step to getting Postfix installed and running.  I do have to admit that it took a few additional troubleshooting steps in my case — most certainly unique to my environment, so I will spare the details.

In order to complete the Postfix implementation, you’ll need to edit several configuration files.  Do not use Notepad for this, as it does not understand the peculiar format of these text files:

One would think that the virtualization layer could also dynamically accommodate for UNIX line formatting, but that evidently was not part of the design.  Fortunately, this is a very minor issue since WordPad can be used without problem:

Once you have the basic configuration in place, restart the service and validate — telnet to port 25 and send a test email.

There you have it, amazing Hyper-V virtualization — same capabilities as VMware vSphere with 25 percent fewer layers!

Related posts:

1. Layers and Layers of FUD
2. Choose any two: Hyper-V, HA, Linux
3. Even GPL can’t make Hyper-V Linux VMs well
4. Does Microsoft really support Linux on Hyper-V?

The author worked extra hard to refute all 9 points — but I do admit being somewhat distracted by the gratuitous use of VMWare throughout.  Evidently, proofreading — unlike immeasurable technology and services discounts — is not one of the benefits extended to guest posters these days, which is a shame.

The Microsoft Virtualization MVP tries to use most of the pages from the playbook, such as memory overcommit FUD:

Relying on a host to overcommit memory to support failover hosts is potentially dangerous and incorrect oversubscription leads to all VMs suffering from performance.

Have you ever had a VM suffer from performance?  I have, and believe me — it’s not pleasant.

But here is a familiar line:

… I would never over commit my VM hosts in production.

Where have we heard that before?   I almost forgot to mention that this guest poster is none other than Chris Steffen — the star of that now-infamous CIO Magazine article who once called VMotion a “gimmick” and said:

I don’t know what kind of environment other than QA or development, or in a staging environment you’d want to do [live migration], but it’s not going to be in any production environment I’m responsible for; it’s not going to be any environment dependent on any kind of SLA or performance requirements. The justification for the inherent risk just isn’t there.

But what does Mr. Steffen say now that Hyper-V has Live Migration?  About what you’d expect — and more:

Also keep in mind that using the System Center suite, the Microsoft solution can Live Migrate VMs to other hosts due to situations that VMware servers cannot even monitor, such as CPU Power, Power Supply Failures, and Fibre Channel congestion.

There are a number of other gems in this convincing piece.  Did you know:

• Hyper-V is more stable than and at least as secure as VMWare [sic]
• vSphere is just as new as Hyper-V

Yep, there is a full-day supply of Kool-Aid in that guest post.

Nice work, Microsoft Virtualization team.  I’m not so sure this sort of thing helps your cause — please keep it up.

Related posts:

1. The Truth About Hyper-V Memory Overcommit
2. Hyper-V [Not Exactly] Dynamic Memory
3. VMware vSphere 5 outperforms Hyper-V by nearly 20%
4. SearchServerMisinformation.com

The Cost Per Application Calculator makes it clear that investing in VMware vSphere 4 significantly reduces your datacenter hardware footprint and associated costs.  Scott Drummonds, the VMware performance expert, recently explained how memory overcommit is the only way to effectively use all of the physical RAM in a hypervisor.

Each time this topic comes up, Microsoft revs up their marketing machine and responds like this:

• Hyper-V is free, use your money to buy RAM upgrades instead of VMware vSphere licenses
• Memory overcommit is risky, VMware tells customers to turn it off anyway

The truth is that Hyper-V will have memory overcommit the moment Microsoft figures it out.  If that day ever does come, watch the messaging quickly change to the familiar, “our customers asked us to implement this…” line.

Why is it fair to make such a bold claim?  Two words:

Dynamic Memory

It’s a little known fact that some of the early releases of Hyper-V R2 actually had a feature that allowed administrators to assign more RAM to virtual machines than was physically present on the host — “Dynamic Memory” a.k.a. memory overcommit.

We can look back and see what bloggers like Mark Wilson had to say:

Microsoft also spoke to me about a dynamic memory capability (just like the balloon model that competitors offer). I asked why the company had been so vocal in downplaying competitive implementations of this technology yet was now implementing something similar and Ward Ralston explained to me that this is not the right solution for everyone but may help to handle memory usage spikes in a VDI environment. Since then, I’ve been advised that dynamic memory will not be in the beta release of Windows Server 2008 R2 and Microsoft is evaluating options for inclusion (or otherwise) at release candidate stage.

And this blog provided more detail on the implementation:

In Hyper-V 1.0, physical memory was hard allocated to the VMs, but in 2.0 the pool of memory is dynamically allocated and removed based VM usage with no service interruption. Dynamically allocating memory to VMs can drastically improve host consolidation rates.

Hyper-V 2.0 VMs are configured with an initial RAM setting (how much the machine boots with) as well as minimum and maximum RAM values. Hyper-V then adds RAM using the Hot-Add function, and removes it using a balloon driver (for supported OSes).

For the visual learners in the crowd, take a look at this VM configuration dialog:

Note that this is nothing like Transparent Page Sharing in VMware ESX — Hyper-V VMs would actually be reconfigured using hot-add memory, so the guest operating systems must cooperate.

In an interview with Steven Bink, even Bob Muglia acknowledged the need for overcommit:

We talked about Vmware ESX and its features like shared memory between VMs, “we definitely need to put that in our product” later [Muglia] said it will be in the next release. Like hot add memory, disk and nic’s will be and Live migration of course, which didn’t make it in this release.

By the way, hot-add memory didn’t make it into Hyper-V R2, either — VMware ESX 4 has it today.

Quick Memory Overcommit?

Evidently, dynamic memory was not even up to the rigorous “Quick” standard and was dropped from the release train.  Perhaps in some future edition of Hyper-V, Quick Memory Overcommit will be offered — with just a few seconds of VM downtime as RAM allocation is dynamically adjusted.  But that’s  just speculation.

Sour Grapes

Instead of finding a way to implement memory overcommit in Hyper-V R2, Microsoft has taken the alternate approach of attacking VMware and declaring the feature unnecessary, unsafe, or too expensive.

The fact is that memory overcommit is an extremely valuable capability and VMware ESX has had it all along.  Some of the Linux-based hypervisors are starting to figure it out.  Until Hyper-V finally adds the feature we’ll continue to hear how easy it is to simply buy more RAM.

How long can you afford to wait?

Related posts:

1. Hyper-V [Not Exactly] Dynamic Memory
2. SearchServerMisinformation.com
3. Hyper-V Dynamic Memory: Not Quite Ready to Demo!
4. Hello Dynamic Memory?