William Lam is the de-facto authority on fully-automated ESXi deployments, and others have provided recipes for configuring VMware ESXi 4.1 installation media for interactive PXE installation.

The good news is that with VMware ESXi 5 it is no longer necessary to create the image.tgz file to accommodate a manual PXE installation, and another handy enhancement to boot.cfg makes it slightly cleaner to organize multiple installation sources on a PXE/TFTP server.

Take a look at the following procedures to learn more:

Prepare the VMware ESXi 5 Installation Media

On your TFTP server, simply extract the contents of the installation ISO into a new directory.

# mkdir esxi50
# mount -o loop VMware-VMvisor-Installer-5.0.0-nnnnnn.x86_64.iso tmp/
# rsync -a tmp/ esxi50/
# chmod +w esxi50/*

That’s it.  No need to create any additional files with ESXi 5.

Modify the ESXi boot.cfg Configuration File

In order to reflect the fact that the installation source is not in the root of the TFTP server, two tweaks are needed.

1. Remove all of the slashes (/) from the boot.cfg file so relative paths will be used (vi :%s#/##g)
2. Add a “prefix” directive to the boot.cfg file to specify the proper subdirectory, from the perspective of the TFTP root

End result will look something like this:

This is much cleaner that repeating the full path dozens of times throughout the file.

Add a PXE Menu Entry

Typically the PXE menu is pxelinux.cfg/default; add the following section and you’ll be set:

LABEL esxi50
 KERNEL /esxi50/mboot.c32
 APPEND -c /esxi50/boot.cfg
 MENU LABEL ESXi-5.0.0

Ready for interactive installation

Now you are ready to boot your physical or virtual machines for an interactive installation of VMware ESXi 5.  It’s very easy to run ESXi 5 inside a VM, now with 64-bit nested guest capabilities!

Related posts:

1. Peaceful Coexistence: WDS and Linux PXE Servers
2. VMware ESXi and Hyper-V installation comparison
3. Create ESXi 4 USB flash drives with Workstation
4. VMware ESXi 4: SD Flash in BL460c G6

VMware vSphere 5 makes it easier than ever to virtualize hypervisor hosts. With new capabilities to run nested 64-bit guests and take snapshots of virtual ESXi VMs, the sky is the limit for your cloud infrastructure development lab. Heck, you can even run Hyper-V on top of vSphere 5 — not that you’d want to.

Physical Host Setup

The physical host running VMware ESXi 5 requires just a few configuration changes; here is a guide:

• Install VMware ESXi 5 on a physical host and configure networking, storage, and other aspects as needed
• Configure a vSwitch and/or Port Group to have Promiscuous Mode enabled
• Create a second Port Group named “Trunk” with VLAN ID All (4095) if you want to use VLANs on virtual hypervisors
• Log in to Tech Support Mode (iLO or ssh) and make the following tweak to enable nested 64-bit guests

  echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

Virtual VMware ESXi Machine (vESXi) Creation

For various reasons, it’s not feasible to clone virtual ESXi VMs. As an alternative, create a fully-configured shell VM to use as a template — it can be cloned before ESXi is installed.

Create a new VM with the following guidance:

• Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
• 2 virtual sockets, 2+ GB RAM
• 4 NICs — connect NIC 1 to the management network and the rest to the “Trunk” network:
• Thin provisioned virtual disks work fine
• Finish creating the VM, then edit the following settings
  • Options/General Options: change Guest Operating System to Other – VMware ESXi 5.x
  • CPU/MMU Virtualization: Use Intel VT … EPT… ( bottom radio button)
• Don’t power this VM on — keep it to act as a template
• Clone and install VMware ESXi via ISO image or PXE boot
• Add to vCenter and configure virtual ESXi hosts for action

Nested 64-bit Guests

With the release of VMware vSphere 5, nested guests can be 64-bit operating systems. Just be sure to make the change to /etc/vmware/config on the physical host as indicated above.

Nested guests can be migrated with vMotion between virtual or physical VMware ESXi hosts; this requires a vMotion network and shared storage.

Nested Hyper-V Virtual Machines

It is possible to run other hypervisors as vSphere virtual machines, and even power on nested VMs. Here you can see Hyper-V running a CentOS virtual machine — all on VMware ESXi. Talk about disrupting the space-time continuum!

A couple of extra tweaks are needed to enable this, and performance is not great. Nevertheless, an amazing feat of engineering from VMware!

Do the following to enable Hyper-V on VMware ESXi:

• Add hypervisor.cpuid.v0 = FALSE to the VM configuration

• Add —-:—-:—-:—-:—-:—-:–h-:—- to the CPU mask for Level 1 ECX (Intel)

For another take, check out William Lam’s post on this topic.

Parting Thoughts

Given the right hardware, it is possible to create a fully-functional VMware test lab that is completely virtual. Go ahead and experiment with the Distributed Virtual Switch, vShield, vCloud Director, and everything else without deploying a ton of servers and storage.

How are you taking advantage of a virtual vSphere environment?

Related posts:

1. VMware ESX 4 can even virtualize itself
2. New VMware ESXi Management Kit
3. Taking snapshots of VMware ESX 4 running in a VM
4. PowerShell Prevents Datastore Emergencies

What’s the difference between ESXi and vSphere?

While that’s an easy one for most VCritical readers to answer, there are newcomers that may benefit from a simple overview.  If you’re here seeking vSphere understanding, welcome!

VMware vSphere Demystified

VMware vSphere is the industry-leading virtualization platform that consists of two primary products: VMware ESXi and vCenter Server.  ESXi is the hypervisor and installs on bare metal hardware.  vCenter Server provides centralized management and allows administrators to configure and monitor ESXi hosts, provision virtual machines, storage, networking, and much more.  The vSphere Client is a Windows application that acts as a single pane of glass to manage either a standalone ESXi host directly or an entire datacenter though vCenter.

VMware ESX vs. ESXi

VMware ESX was introduced a decade ago and will be discontinued with the upcoming major release.  Carrying the torch forward will be ultra-slim VMware ESXi, which has already seen several years of successful production deployments.  Both products are bare-metal hypervisors — they install directly onto a server instead of a traditional general purpose operating system — and have the same capabilities, accommodating any licensed feature from Essentials to Enterprise Plus: vMotion, DRS, HA, FT, and more.

The primary difference is that with ESXi the Red Hat Linux service console is gone, leaving just the hypervisor and critical supporting features.  By eliminating tons of unnecessary Linux components, ESXi footprint is measured in mere megabytes — not gigabytes like competitors.

Microsoft Hyper-V Server vs. VMware ESXi

I always get a kick out of well-meaning folks that try to claim that ESXi should not be compared to Hyper-V on Windows Server 2008 R2 SP1, insisting that it’s much closer to compare with Microsoft Hyper-V Server.  Hyper-V Server, for those less familiar, is a free product that is essentially Windows Server 2008 Core with the Hyper-V role and a snazzy text-based menu that allows you to do a few key things — like run Windows Update.

Make no mistake about it, weighing in at several gigabytes and requiring care and feeding on Patch Tuesdays, Hyper-V Server is anything but a thin, purpose-built hypervisor.  It’s Windows — just not the “Windows you know.”  As Aidan Finn, Microsoft MVP and Hyper-V expert says, “…I would almost never install Server Core…”

Another vSphere Advantage over Microsoft Virtualization

Readers of  VCritical have learned many of the technical advantages that ESX/i has over Hyper-V, but there is another advantage that vSphere holds over Microsoft when it comes to virtual infrastructure:  By releasing both the hypervisor and the advanced management in lockstep, VMware vSphere is a platform that is greater than the sum of its parts.

You might be wondering, how a virtualization platform could introduce new, advanced capabilities if the hypervisor and management products are on different release cycles.  No need to wonder, just witness Microsoft virtualization and see firsthand.  Don’t take my word for it — look at how a real Hyper-V customer sees things:

It’s unbelievable that Microsoft would roll out new features to the virtualization product that are not supported in their virtualization management product… The Hyper-V and SCVMM teams don’t march in lockstep, and what updates they do release are too far apart.

Clearly Hyper-V customers have not been pleased with the manageability delays, but that’s not all…

The Bad News Flows in Two Directions

Later this year Microsoft will release System Center Virtual Machine Manager 2012, but the next version of Windows Server and Hyper-V is still merely the subject of speculation.  This leapfrog release cadence creates a situation where VMM 2012 must attempt to overcome limitations with the platform by painting over them, in lieu of elegant solutions integrated with lower layers of infrastructure.

Consider this example:  Instead of introducing a new streamlined hypervisor clustering capability, VMM 2012 attempts to orchestrate the 29 steps currently required to install, configure, and validate a failover cluster for use with virtual machines.  Maybe some administrators will find it an improvement over the existing manual effort, but is it better than the simple drag-and-drop design found in vSphere?

Synchronized vs. Staggered

VMware vSphere is the leading virtualization platform for many reasons.  It’s hard to understand how asynchronous hypervisor and management releases would be acceptable, let alone desirable.  Just another compromise encountered when building a hypervisor on top of  a general purpose operating system.

Related posts:

1. What is VMware ESXi?
2. VMware ESXi 5 Scripted Install to USB Flash
3. If VMware ESXi 4 is so small, why is it so big?
4. VMware ESXi and Hyper-V installation comparison

Power on and SSH to a Linux VM from Mac OS X

Consider this real-world situation:  You are working remotely on your Mac and need to log into a powered-off Linux VM back in the datacenter.  Don’t fret — if you can ssh to the host using Tech Support Mode (VMware ESXi 4.1 shown here), you are most of the way there.  Thanks to vim-cmd you can find your VM, power it on, and learn the guest OS IP address in no time:

Let’s break it down step by step:

1. Use grep to find the ID of your VM (1008 in this example):

~ # vim-cmd vmsvc/getallvms | grep -i CENTOS1
1008   CENTOS1      [CX4-01-LUN9] CENTOS1/CENTOS1.vmx         centos64Guest

2. Quick sanity check to verify the VM is actually powered off:

~ # vim-cmd vmsvc/get.summary 1008 | egrep '(power|ip)'
      powerState = "poweredOff",
      ipAddress = ,

3. Power on, wait a few moments, and check status again:

~ # vim-cmd vmsvc/power.on 1008
Powering on VM:
~ # vim-cmd vmsvc/get.summary 1008 | egrep '(power|ip)'
      powerState = "poweredOn",
      ipAddress = "10.136.234.38",
~ #

4.  Connect with ssh (or RDP, if targeting a Windows VM) and get on with things

Enjoy!

Related posts:

1. VI Client SSL thumbprint cache cleanup
2. Easily view guest IP addresses with vSphere Client
3. Exclusive: Hyper-V 2013 Product Roadmap
4. Idle RHEV Hypervisors save power?

I just found another point of view on hypervisor footprint size — take a look at this excerpt:

Small footprint: any features not applicable to our specific … scenarios are removed.  This guarantees that we do not have to worry about updating or fixing unnecessary code, meaning less churning or required reboots for the host.  All critical code paths are also highly optimized for our … scenarios.

Any guesses where that came from?  Must be more of that VMware FUD!

Actually, it was one of the Windows Azure design principles.  Which makes sense if you think about it — vSphere, with small-footprint ESXi,  is the perfect foundation for cloud computing.

Related posts:

1. Hyper-V Server Hyperbole
2. A very flashy hypervisor: Hyper-V Server R2
3. Pimp My Hypervisor
4. Microsoft seeks to stem Azure exodus with huge appliance

Check out the details and get scripting!

Related posts:

1. VMware ESXi 4: SD Flash in BL460c G6
2. If VMware ESXi 4 is so small, why is it so big?
3. New VMware ESXi Management Kit
4. Win a MacBook Air or iPod Touch

In my HP BladeSystem c7000 enclosure with BL460 G6 (Nehalem) blades, the SD slot is easily exposed by sliding each blade out of the chassis:

Once the SD flash card is in place, it is trivial to install VMware ESXi:

You may also be interested in articles from Rick Scherer and TechHead on this topic.

Hyper-V on Flash?

Sure, Hyper-V Server — but not Windows Server 2008 with Hyper-V — supports boot-from-flash when purchased from a server OEM, but to the best of my knowledge there are no vendors shipping such configurations at this time.  Note that Microsoft recommends a 16GB flash drive for Hyper-V Server deployment (compared to a slender 1GB for ESXi 4), but a PlayStation Portable is an acceptable substitute in a pinch.

Motivation

Two great reasons for running ESXi from flash:

• Eliminate the need for local disks — lower power and cooling requirements
• Provide dual-boot capabilities in a lab environment

Why would you run VMware ESXi from flash?

Related posts:

1. Create ESXi 4 USB flash drives with Workstation
2. VMware ESXi 5 Scripted Install to USB Flash
3. A very flashy hypervisor: Hyper-V Server R2
4. The VMware ESXi 4 64MB Hypervisor Challenge

The case Microsoft is really trying to make here is that if you don’t have VMware virtualization, then you don’t have VMware.  True enough, I suppose.   But you still have the same number of layers.

I took the liberty of making some corrections to the Microsoft collateral and am pleased to present three more appropriate ways to depict these virtualization layers.

The Objective Comparison

Truth be told, both virtualization platforms have approximately the same high-level depiction of layers:

Nothing surprising there — that is how any truly technical content would describe the arrangement.  Notice that virtualization is actually not above the application layer.

The Proportional Correction

VMware ESX and Microsoft Hyper-V both function as a hypervisor, but it turns out that since ESX has a much smaller footprint, one might envision the stack of colored boxes more like this:

Combining Layers — The Right Way

Is virtualization more like hardware or more like software?  If I had to choose, I would pick hardware.  Perhaps those are the two layers that should be combined, like this:

As long as the hypervisor is not tied to a general-purpose operating system, this makes good sense.  However, since Hyper-V is merely a role in Windows Server, it would be a real stretch to picture the same combination for Microsoft virtualization.  Now that I’ve reasoned it through, maybe three layers isn’t so bad after all.

How do you like them layers?

Related posts:

1. Pimp My Hypervisor
2. Linux apps run directly on Windows with virtualization!
3. Would you buy a hypervisor from these guys?
4. A very flashy hypervisor: Hyper-V Server R2

Requirements

• The VMware ESXi 4 dd image file (Vladan’s site may help)
• A 64MB flash drive — check your junk drawer
• A Linux machine — make sure you have mkfs.vfat
• This fdisk script — save it as a txt file

Extract necessary components from the large image

Use dd to copy the first 1.5MB of the boot partition and the files that make up ESXi.

# dd if=VMware-VMvisor-big-175625-x86_64.dd of=esxi-boot-part skip=32 count=3000
3000+0 records in
3000+0 records out
1536000 bytes (1.5 MB) copied, 0.036249 seconds, 42.4 MB/s
# mkdir tmp
# mkdir esxi
# mount -o loop,offset=$((8224*512)) VMware-VMvisor-big-175625-x86_64.dd tmp/
# cp tmp/* esxi
# sync
# umount tmp/

Prepare the flash device

Plug in your flash device and use dmesg to determine the device id (/dev/sdk in this example).  Run fdisk on it and delete all partitions – very important.  Then, run my fdisk script to create the required partition structure.  If this doesn’t work on your flash drive due to different size/geometry, you will need to create the partitions yourself — refer to the table at the end of these instructions for reference.

# fdisk /dev/sdk < fdisk.txt

Write data to the flash device

Use dd to write the saved boot partition to flash.

# dd if=esxi-boot-part of=/dev/sdk4
3000+0 records in
3000+0 records out
1536000 bytes (1.5 MB) copied, 1.31462 seconds, 1.2 MB/s
# sync

Then create a FAT filesystem on the flash drive, mount it, and copy the ESXi files.

# mkfs.vfat /dev/sdk5
mkfs.vfat 2.11 (12 Mar 2005)
# mount /dev/sdk5 tmp/
# cp esxi/* tmp/
# sync
# umount tmp/

Double-check your partition layout

If your disk partition does not look like this, something went wrong.

# fdisk -l /dev/sdk
 
Disk /dev/sdk: 65 MB, 65273856 bytes
3 heads, 42 sectors/track, 1011 cylinders
Units = cylinders of 126 * 512 = 64512 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdk1              26        1011       62118    5  Extended
/dev/sdk4   *           1          25        1554    4  FAT16 <32M
/dev/sdk5              26        1011       62097    6  FAT16

Test it out

If you followed all of the instructions above, you should be able to boot from that flash drive.

There you have it — a fully functional VMware ESXi 4 system entirely contained on a 64MB flash device. Of course, this procedure is entirely unsupported, so please do not use a system like this in production. Please!

Related posts:

1. Create ESXi 4 USB flash drives with Workstation
2. If VMware ESXi 4 is so small, why is it so big?
3. VMware ESXi 5 Interactive PXE Installation Improvements
4. VMware ESXi 4: SD Flash in BL460c G6

The flagship hypervisor from VMware actually comes in two different form-factors: classic VMware ESX with a Linux-based management console and thin VMware ESXi that can boot from a small embedded flash drive.  Even though ESXi is available for free, it is functionally equivalent to classic ESX when fully licensed and managed by vCenter Server.

VMware ESX 3i, as it was originally known in the VI3 era, was just a tiny 32MB.  With VMware vSphere 4, the name changed slightly to VMware ESXi 4, and grew to about 60MB — still extremely small for a fully-functional hypervisor with HA, VMotion, DRS, and all the other vSphere features.

While ESXi is less than 60MB, installing it actually requires a 1GB flash device.  This raises a question:

VMware claims that ESXi is small — why does it require so much storage?

Unlike a general purpose operating system, ESXi was designed to be deployed as an image — similar to a router.  The “installer” merely paves your empty storage device with some partitions.  In fact, there are two separate partitions created for ESXi images — this allows administrators to roll back to a known-working environment if an update is not successful.  In order to be prepared for the future, these two partitions are several times larger than the data they contain.

Not only does a 1GB flash device contain the ESXi hypervisor, it also provides VMware Tools for various supported operating systems and a copy of the vSphere Client which administrators can download and install to their workstations.  These components are not executed by the hypervisor at all — they can be obtained through other means, but it is very convenient to have them right on the host.

Take a look at this visual to better understand the partition layout:

• Partition 4 enables the actual booting from flash
• Partition 5 contains files that make up the VMware ESXi image
• Partition 6 is used to store a future update to the image file
• Partition 7 is for a core dump
• Partition 8 contains tools, drivers, and the Windows-based vSphere Client installer

In case you are curious, here are the contents of that last partition:

As you can see, it contains a bunch of Windows executables and ISO images for your various guests.

Still not convinced?

Some may still doubt the VMware ESXi 4 60MB footprint claim, but one sure way to make the case is to simply run ESXi from a very small flash drive.  Olivier Cremel, the inventor of VMware ESXi, explained to me how to do just that. In my next post on this topic I’ll walk you through it, step by step.  Don’t miss out — subscribe to the VCritical RSS feed today!

Related posts:

1. The VMware ESXi 4 64MB Hypervisor Challenge
2. Create ESXi 4 USB flash drives with Workstation
3. VMware ESXi 4: SD Flash in BL460c G6
4. VMware ESXi 5 Scripted Install to USB Flash