Windows deserves applause for reliability improvements in recent years.  Unfortunately, the most reliable Windows design will never be able to counteract the damage that can be inflicted by a misbehaving device driver.  In fact, take a look at this slide from a Mark Russinovich TechEd session where he makes the point that the majority of Windows blue screens (BSODs) are caused by third-party drivers:

Experts agree — Windows reliability is a function of driver reliability.

What about those drivers included with Windows?

You might be under the impression that these third-party drivers are for off-brand NICs purchased from the clearance bin at Fry’s.  That’s not exactly the case — plenty of drivers are included right on the Windows DVD.

Consider this recent situation detailed by Mark Wilson.  He set up a Hyper-V test machine with some Intel PRO/100 NIC cards and when he plugged in an Ethernet cable — instant BSOD!

Mark’s conclusion:

It’s good to know that Hyper-V was not at fault here: sure, it shows that a rogue device driver can bring down a Windows system but that’s hardly breaking news…

Meanwhile, back at the datacenter

Microsoft Hyper-V poster child Crutchfield recently found that the Broadcom NICs embedded in every one of their Dell servers were randomly losing virtual switch bindings after upgrading to Hyper-V R2.  Hello downtime!

The administrator left us with the following thoughts:

Currently, Microsoft doesn’t have a work around for this issue and nor is there an ETA for resolution. After all, it may not be Microsoft, but could very well be Broadcom.

So, what do we do? Well, in my configurations I can’t afford these little gotchas and I will be working only with my trusted Intel NIC’s for my virtual networks.

Uh, you might want to check with Mark Wilson first about Intel drivers and Hyper-V.

The chain breaks when the weakest link fails

Notice the consistent theme in these two scenarios?  The administrators are not blaming Hyper-V, instead focusing on the third-party hardware and drivers.  Folks, it just does not matter — a hypervisor failure is incredibly disruptive.

When selecting the foundation for your private cloud, will you choose a hypervisor based on a general-purpose OS that aims to support hundreds of hardware variations from across the spectrum?  Or will you build on a rock-sold, purpose-built platform that is specifically designed to aggregate and pool resources in your datacenter?

Related posts:

1. Yes, NIC teaming is not unsupported
2. Hyper-V Linux integration components no longer Connected
3. The Million Hypervisor March
4. Of Mice and Xen

“Yo dawg, I heard you like Patch Tuesday, so I put Windows in your hypervisor so you can patch it while you virtualize!”

Inspired by a joke from Kendrick Coleman. With apologies to Xzibit.

Related posts:

1. Lighten Up
2. Let them eat cake!
3. A very flashy hypervisor: Hyper-V Server R2
4. Linux apps run directly on Windows with virtualization!

One outstanding achievement is the way the Hyper-V architects managed to completely collapse the guest operating system layer directly into the hypervisor.  Take a look at this diagram from a Microsoft Virtualization competitive brochure:

[Yes, it's that same brochure again.]

There are several key differences between Microsoft and VMware hypervisor architectures.  While the hardware and applications layers are pretty straightforward, applications don’t run directly on hardware.  As you are about to see, these two platforms take very different approaches when it comes to virtualizing workloads.

The VMware vSphere solution is to use a thin, high-performance virtualization layer that enables a single physical system to simultaneously host multiple virtual machines.  Each of these virtual machines executes isolated instances of traditional guest operating systems — Windows, Linux, Solaris, Netware, etc.

Due to apparent technological advancements in Hyper-V, it is now possible to run applications of all kinds directly on that integrated Windows OS/hypervisor layer without the need for a guest operating system at all.  This allows Microsoft to omit an entire layer from their virtualization architecture.

I’m sure you have seen various announcements about the exceptional support that Hyper-V has for Linux workloads.  Without a guest operating system layer, one must logically conclude that Linux applications are actually running on the Windows/Hyper-V hypervisor integration directly.  Spectacular!

I thought I’d test this technological breakthrough in the lab myself by running Postfix, a very popular Linux SMTP mail server, on Windows Server 2008 R2 with Hyper-V.  According to the diagram above, Postfix is classified as an application — the third layer.

Now, in order to install Postfix, I have to interact with the second layer in the stack, which is the “Microsoft Operating System w/Virtualization.”  Since I’m an RHCE, I decided to use a Red Hat Enterprise Linux package of Postfix, but we can all agree that SUSE Linux Enterprise Server 11 would work equally well.  However, I would not recommend using Ubuntu packages since that distribution is not supported by Hyper-V.  Another side benefit to this technique is the fact that Windows has great mouse support.

Just as one would do on a legacy Linux system, I opened a trusty command shell and installed the package like so:

This is the first step to getting Postfix installed and running.  I do have to admit that it took a few additional troubleshooting steps in my case — most certainly unique to my environment, so I will spare the details.

In order to complete the Postfix implementation, you’ll need to edit several configuration files.  Do not use Notepad for this, as it does not understand the peculiar format of these text files:

One would think that the virtualization layer could also dynamically accommodate for UNIX line formatting, but that evidently was not part of the design.  Fortunately, this is a very minor issue since WordPad can be used without problem:

Once you have the basic configuration in place, restart the service and validate — telnet to port 25 and send a test email.

There you have it, amazing Hyper-V virtualization — same capabilities as VMware vSphere with 25 percent fewer layers!

Related posts:

1. Choose any two: Hyper-V, HA, Linux
2. Even GPL can’t make Hyper-V Linux VMs well
3. Which guest operating systems can be customized by SCVMM?
4. Hyper-V Linux integration components no longer Connected

I just found another point of view on hypervisor footprint size — take a look at this excerpt:

Small footprint: any features not applicable to our specific … scenarios are removed.  This guarantees that we do not have to worry about updating or fixing unnecessary code, meaning less churning or required reboots for the host.  All critical code paths are also highly optimized for our … scenarios.

Any guesses where that came from?  Must be more of that VMware FUD!

Actually, it was one of the Windows Azure design principles.  Which makes sense if you think about it — vSphere, with small-footprint ESXi,  is the perfect foundation for cloud computing.

Related posts:

1. A very flashy hypervisor: Hyper-V Server R2
2. Hyper-V Server Hyperbole
3. If VMware ESXi 4 is so small, why is it so big?
4. Pimp My Hypervisor

A screenshot of Dynamic Memory configuration in a post-RTM build of Windows Server 2008 R2 has surfaced in an article at Softpedia.

Listen… Do you hear that?

It’s the sound of Microsoft Virtualization curbing their criticism of memory overcommit.

Hat tip to Aidan Finn.

Related posts:

1. The Truth About Hyper-V Memory Overcommit
2. Hyper-V More Stable, Just as Mature as VMware vSphere?
3. InformationWeek: Hyper-V Not For Enterprise
4. SCVMM Damage Control

The author worked extra hard to refute all 9 points — but I do admit being somewhat distracted by the gratuitous use of VMWare throughout.  Evidently, proofreading — unlike immeasurable technology and services discounts — is not one of the benefits extended to guest posters these days, which is a shame.

The Microsoft Virtualization MVP tries to use most of the pages from the playbook, such as memory overcommit FUD:

Relying on a host to overcommit memory to support failover hosts is potentially dangerous and incorrect oversubscription leads to all VMs suffering from performance.

Have you ever had a VM suffer from performance?  I have, and believe me — it’s not pleasant.

But here is a familiar line:

… I would never over commit my VM hosts in production.

Where have we heard that before?   I almost forgot to mention that this guest poster is none other than Chris Steffen — the star of that now-infamous CIO Magazine article who once called VMotion a “gimmick” and said:

I don’t know what kind of environment other than QA or development, or in a staging environment you’d want to do [live migration], but it’s not going to be in any production environment I’m responsible for; it’s not going to be any environment dependent on any kind of SLA or performance requirements. The justification for the inherent risk just isn’t there.

But what does Mr. Steffen say now that Hyper-V has Live Migration?  About what you’d expect — and more:

Also keep in mind that using the System Center suite, the Microsoft solution can Live Migrate VMs to other hosts due to situations that VMware servers cannot even monitor, such as CPU Power, Power Supply Failures, and Fibre Channel congestion.

There are a number of other gems in this convincing piece.  Did you know:

• Hyper-V is more stable than and at least as secure as VMWare [sic]
• vSphere is just as new as Hyper-V

Yep, there is a full-day supply of Kool-Aid in that guest post.

Nice work, Microsoft Virtualization team.  I’m not so sure this sort of thing helps your cause — please keep it up.

Related posts:

1. The Truth About Hyper-V Memory Overcommit
2. On Quick Storage Migration in R2
3. Hello Dynamic Memory?
4. How to see if Quick Migration is right for your workload

The Cost Per Application Calculator makes it clear that investing in VMware vSphere 4 significantly reduces your datacenter hardware footprint and associated costs.  Scott Drummonds, the VMware performance expert, recently explained how memory overcommit is the only way to effectively use all of the physical RAM in a hypervisor.

Each time this topic comes up, Microsoft revs up their marketing machine and responds like this:

• Hyper-V is free, use your money to buy RAM upgrades instead of VMware vSphere licenses
• Memory overcommit is risky, VMware tells customers to turn it off anyway

The truth is that Hyper-V will have memory overcommit the moment Microsoft figures it out.  If that day ever does come, watch the messaging quickly change to the familiar, “our customers asked us to implement this…” line.

Why is it fair to make such a bold claim?  Two words:

Dynamic Memory

It’s a little known fact that some of the early releases of Hyper-V R2 actually had a feature that allowed administrators to assign more RAM to virtual machines than was physically present on the host — “Dynamic Memory” a.k.a. memory overcommit.

We can look back and see what bloggers like Mark Wilson had to say:

Microsoft also spoke to me about a dynamic memory capability (just like the balloon model that competitors offer). I asked why the company had been so vocal in downplaying competitive implementations of this technology yet was now implementing something similar and Ward Ralston explained to me that this is not the right solution for everyone but may help to handle memory usage spikes in a VDI environment. Since then, I’ve been advised that dynamic memory will not be in the beta release of Windows Server 2008 R2 and Microsoft is evaluating options for inclusion (or otherwise) at release candidate stage.

And this blog provided more detail on the implementation:

In Hyper-V 1.0, physical memory was hard allocated to the VMs, but in 2.0 the pool of memory is dynamically allocated and removed based VM usage with no service interruption. Dynamically allocating memory to VMs can drastically improve host consolidation rates.

Hyper-V 2.0 VMs are configured with an initial RAM setting (how much the machine boots with) as well as minimum and maximum RAM values. Hyper-V then adds RAM using the Hot-Add function, and removes it using a balloon driver (for supported OSes).

For the visual learners in the crowd, take a look at this VM configuration dialog:

Note that this is nothing like Transparent Page Sharing in VMware ESX — Hyper-V VMs would actually be reconfigured using hot-add memory, so the guest operating systems must cooperate.

In an interview with Steven Bink, even Bob Muglia acknowledged the need for overcommit:

We talked about Vmware ESX and its features like shared memory between VMs, “we definitely need to put that in our product” later [Muglia] said it will be in the next release. Like hot add memory, disk and nic’s will be and Live migration of course, which didn’t make it in this release.

By the way, hot-add memory didn’t make it into Hyper-V R2, either — VMware ESX 4 has it today.

Quick Memory Overcommit?

Evidently, dynamic memory was not even up to the rigorous “Quick” standard and was dropped from the release train.  Perhaps in some future edition of Hyper-V, Quick Memory Overcommit will be offered — with just a few seconds of VM downtime as RAM allocation is dynamically adjusted.  But that’s  just speculation.

Sour Grapes

Instead of finding a way to implement memory overcommit in Hyper-V R2, Microsoft has taken the alternate approach of attacking VMware and declaring the feature unnecessary, unsafe, or too expensive.

The fact is that memory overcommit is an extremely valuable capability and VMware ESX has had it all along.  Some of the Linux-based hypervisors are starting to figure it out.  Until Hyper-V finally adds the feature we’ll continue to hear how easy it is to simply buy more RAM.

How long can you afford to wait?

Related posts:

1. Hello Dynamic Memory?
2. The Truth About Storage Hot Add and Remove
3. Exclusive: Hyper-V 2013 Product Roadmap
4. Hyper-V More Stable, Just as Mature as VMware vSphere?

The author, Elias Khnaser, did an excellent job of articulating several of the key advantages that VMware vSphere has over Hyper-V — memory overcommit, hypervisor footprint, maturity, and more.

While you may already enjoy the numerous advantages of VMware ESX virtualization and purpose-built vCenter management, perhaps you have a colleague — or boss — who would benefit from reading the article.  Pass it on!

Related posts:

1. CIO deploys Hyper-V, loses job to 15-year-old
2. vCenter for Linux article disappointing
3. Hey boss, where do you want these virtual machines?
4. Hyper-V More Stable, Just as Mature as VMware vSphere?

Virtualization management tasks such as cloning existing virtual machines or deploying new ones from templates often involve slinging around multi-gigabyte chunks of data.  It’s preferable to move or copy data on the SAN instead of the LAN because it can be faster, but more importantly, doing so reduces impact to other services that rely on the LAN.

Use the SAN, ESX!

VMware ESX will transfer VMs and templates over a SAN connection whenever possible but, if necessary, it also does a fine job of moving those bits over a standard LAN.  This goes for VMs as well as templates because with VMware vSphere, templates live on SAN datastores, too — shared among multiple ESX hosts — not on network file shares.  If templates were stored on a file server, there would be no choice other than to copy those multi-gigabyte files over the LAN and potentially impact other production traffic.

Nice design choice, VMware!  [Actually, if you go back to the early days of VirtualCenter 1.x, templates could also be stored directly on the VC server and deployed over the network.  That option was removed as of VC 2.0.]

So far, so good.  Let’s take a look at how Hyper-V stacks up.

Obfuscating the Inequalities

In the tradition of the recent storage hot add/remove claims, Microsoft has again gone the extra mile to give the appearance of feature parity.

Take a look at this excerpt from the System Center Virtual Machine Manager 2008 R2 What’s New page:

SAN migration into and out of clustered hosts: This allows virtual machines to migrate into and out of clusters using a SAN transfer, which saves the time required for copying the virtual machine file over the network.

And the features page is even more dramatic, proclaiming:

Virtual machine images can be large and difficult to move over the network. VMM auto-detects SAN infrastructure and enables copying of virtual machine images over fiber at fast speeds, thus leveraging SAN investments.

Sounds good, just like VMware — and at one-sixth the price!  But there is a small problem: it is not true.

Hyper-V uses the BITS service to transfer VMs over the LAN in almost all cases.  There is a scenario where SCVMM can orchestrate the disconnection and reconnection of a SAN LUN, “transferring” a VM to another host.  Enabling the feature requires additional configuration and some software from your SAN vendor.  Of course, it also means going back to one VM per LUN and foregoing the wonders of CSV.  Any takers?

The SCVMM Library Server

SCVMM 2008 R2 provides VM template functionality for Hyper-V.  Templates  and ISO images are stored in a Library, which uses standard Windows file services.  Obviously, that means no SAN copying for template deployments, either — kind of like VMware VirtualCenter 1.0.

Conclusion

“SAN LUN disconnect and reconnect” just doesn’t have the same ring to it as “SAN transfer” but I’m not sure that is sufficient justification for these misleading marketing claims.  And to use the word “copying” is simply dishonest.

Don’t make the mistake of assuming Hyper-V is just like VMware ESX.  Compare for yourself — seeing is believing.

Related posts:

1. IGT Part 2: SAN Misers
2. Storage vendors unanimously applaud SCVMM innovation
3. The Truth About Storage Hot Add and Remove
4. Using SCVMM to attach ISO images to VMware ESX VMs

• vStorage VMFS Volume Grow
• Hot Extend for Virtual Disks
• Hot Virtual Disk Add/Remove

With these capabilities, if space gets tight in your vSphere environment, it is easy to be proactive and address the issue before anyone notices.  The process goes something like this:

• Allocate additional physical hard disk space on your SAN to an appropriate LUN
• Grow your VMFS datastore onto the newly added free space
• Extend, or add new, virtual disks for the VMs that need more storage
• Expand the volume inside the guest to create more usable space

All without a reboot: zero downtime.

Not All Virtualization Platforms are Created Equal

Don’t jump to the conclusion that all hypervisors offer the same flexibility.  Perhaps you are wondering about Hyper-V capabilities?

First, let’s take a look at some Microsoft System Center Virtual Machine Manager 2008 R2 (SCVMM) marketing statements:

The What’s New page announces:

Hot addition/removal of Storage: Allows the addition and removal of storage to virtualized infrastructure without interruption. Additionally, “live” management of virtual hard disk (VHDs) or iSCSI pass through disks, allows administrators to take advantage of additional backup scenarios and readily use mission critical and storage-intensive applications.

The Top Benefits list proclaims:

Hot addition/removal of storage: With this capability, administrators can quickly and efficiently respond to changing storage requirements of virtual machines. This ability to hot-add additional storage eliminates the previous need to take the host down to upgrade storage thus increasing business continuity for end users and reducing complexity for administrators. Additionally it allows administrators to confidently deploy mission critical applications (in which up-time is of paramount importance) that may have rapidly changing storage requirements such as web, database or other business applications.

An IT decision-maker just might get the impression that both ESX and Hyper-V have essentially the same features. They do not.

You may be surprised to find out that all of the descriptions above merely refer to adding a new virtual disk to a VM — providing the conditions are right.  You cannot grow an existing VHD, and you can’t safely remove a VHD with SCVMM.

Plan Ahead

First things first.  If your Hyper-V VM does not have a virtual SCSI adapter — templates and VMs from Hyper-V R1 do not — you won’t be able to hot add a new VHD until you correct that shortcoming.  Hello downtime.

SCVMM can add a new blank virtual disk to a VM or it can copy an existing one across the network from the Library — if you copied it there beforehand, but there is no way to add a VHD that may already be present on your SAN — even if it is already sitting right next to the destination VM.

Removal?

Up-time may be of paramount importance, but preventing data-loss was evidently not part of the original design.  Removing a VHD with SCVMM results in the immediate deletion of the underlying VHD file.  Ouch!  Thankfully, a recent patch improves administrator job security by throwing up a warning before this happens, providing an option to cancel.  There is no way to simply disconnect a VHD using SCVMM.

It turns out that if you really want to take advantage of those “Additional Backup Scenarios” by hot adding and removing virtual disks, you need your trusty Hyper-V Manager utility.  Still think System Center is a single pane of glass?  And that’s not the only task that requires administrators to switch between Hyper-V interfaces.

Conclusion

Microsoft is trying hard to ride the coattails of VMware ESX.  The latest release of Hyper-V R2 still does not have capabilities enjoyed by VMware administrators since the ESX 3.5 days.

Don’t believe the obfuscated marketing literature.  VMware vSphere is for real.

Related posts:

1. vSphere Thin-Provisioned Disk Performance
2. Responsible Thin Provisioning in VMware vSphere
3. The Truth About VM SAN Transfers
4. Easy recovery from a full VMware ESX datastore