Now with VMware ESXi 5, it is possible to perform an automated installation to USB storage.  Simply specify usb-storage as part of the install command in an automated install script.  For example:

install --firstdisk=usb-storage --overwritevmfs

For more flexibility in a single script, multiple disk types can be specified; they will be tried in order.  Below is an example install command from a script for HP servers.  If USB flash is not installed, the on-board SAS disks will be used.  If neither are present — such as when installing virtual ESXi systems — generic local storage is the last resort.

install --firstdisk=usb-storage,hpsa,local --overwritevmfs

Perhaps not one of the major new feature of VMware vSphere 5, but handy.

Boot Hyper-V From Flash?

You might not know this, but Microsoft Hyper-V has had the ability to boot from flash as well — announced two years ago, but yet to be acknowledged in an actual production deployment.  Of course, this bleeding-edge configuration option is only supported if installed by OEMs, not end-users.  Anyone considering Hyper-V for virtualization is highly encouraged to give it a try; it’s a great opportunity to learn a lot about Windows registry manipulation and command line tools such as imagex and diskpart.

But seriously, VMware vSphere 5 will be here soon, and you’re going to love it.

Related posts:

1. Create ESXi 4 USB flash drives with Workstation
2. VMware ESXi 4: SD Flash in BL460c G6
3. The VMware ESXi 4 64MB Hypervisor Challenge
4. A very flashy hypervisor: Hyper-V Server R2

William Lam is the de-facto authority on fully-automated ESXi deployments, and others have provided recipes for configuring VMware ESXi 4.1 installation media for interactive PXE installation.

The good news is that with VMware ESXi 5 it is no longer necessary to create the image.tgz file to accommodate a manual PXE installation, and another handy enhancement to boot.cfg makes it slightly cleaner to organize multiple installation sources on a PXE/TFTP server.

Take a look at the following procedures to learn more:

Prepare the VMware ESXi 5 Installation Media

On your TFTP server, simply extract the contents of the installation ISO into a new directory.

# mkdir esxi50
# mount -o loop VMware-VMvisor-Installer-5.0.0-nnnnnn.x86_64.iso tmp/
# rsync -a tmp/ esxi50/
# chmod +w esxi50/*

That’s it.  No need to create any additional files with ESXi 5.

Modify the ESXi boot.cfg Configuration File

In order to reflect the fact that the installation source is not in the root of the TFTP server, two tweaks are needed.

1. Remove all of the slashes (/) from the boot.cfg file so relative paths will be used (vi :%s#/##g)
2. Add a “prefix” directive to the boot.cfg file to specify the proper subdirectory, from the perspective of the TFTP root

End result will look something like this:

This is much cleaner that repeating the full path dozens of times throughout the file.

Add a PXE Menu Entry

Typically the PXE menu is pxelinux.cfg/default; add the following section and you’ll be set:

LABEL esxi50
 KERNEL /esxi50/mboot.c32
 APPEND -c /esxi50/boot.cfg
 MENU LABEL ESXi-5.0.0

Ready for interactive installation

Now you are ready to boot your physical or virtual machines for an interactive installation of VMware ESXi 5.  It’s very easy to run ESXi 5 inside a VM, now with 64-bit nested guest capabilities!

Related posts:

1. Peaceful Coexistence: WDS and Linux PXE Servers
2. VMware ESXi and Hyper-V installation comparison
3. Create ESXi 4 USB flash drives with Workstation
4. VMware ESXi 4: SD Flash in BL460c G6

VMware vSphere 5 makes it easier than ever to virtualize hypervisor hosts. With new capabilities to run nested 64-bit guests and take snapshots of virtual ESXi VMs, the sky is the limit for your cloud infrastructure development lab. Heck, you can even run Hyper-V on top of vSphere 5 — not that you’d want to.

Physical Host Setup

The physical host running VMware ESXi 5 requires just a few configuration changes; here is a guide:

• Install VMware ESXi 5 on a physical host and configure networking, storage, and other aspects as needed
• Configure a vSwitch and/or Port Group to have Promiscuous Mode enabled
• Create a second Port Group named “Trunk” with VLAN ID All (4095) if you want to use VLANs on virtual hypervisors
• Log in to Tech Support Mode (iLO or ssh) and make the following tweak to enable nested 64-bit guests

  echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

Virtual VMware ESXi Machine (vESXi) Creation

For various reasons, it’s not feasible to clone virtual ESXi VMs. As an alternative, create a fully-configured shell VM to use as a template — it can be cloned before ESXi is installed.

Create a new VM with the following guidance:

• Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
• 2 virtual sockets, 2+ GB RAM
• 4 NICs — connect NIC 1 to the management network and the rest to the “Trunk” network:
• Thin provisioned virtual disks work fine
• Finish creating the VM, then edit the following settings
  • Options/General Options: change Guest Operating System to Other – VMware ESXi 5.x
  • CPU/MMU Virtualization: Use Intel VT … EPT… ( bottom radio button)
• Don’t power this VM on — keep it to act as a template
• Clone and install VMware ESXi via ISO image or PXE boot
• Add to vCenter and configure virtual ESXi hosts for action

Nested 64-bit Guests

With the release of VMware vSphere 5, nested guests can be 64-bit operating systems. Just be sure to make the change to /etc/vmware/config on the physical host as indicated above.

Nested guests can be migrated with vMotion between virtual or physical VMware ESXi hosts; this requires a vMotion network and shared storage.

Nested Hyper-V Virtual Machines

It is possible to run other hypervisors as vSphere virtual machines, and even power on nested VMs. Here you can see Hyper-V running a CentOS virtual machine — all on VMware ESXi. Talk about disrupting the space-time continuum!

A couple of extra tweaks are needed to enable this, and performance is not great. Nevertheless, an amazing feat of engineering from VMware!

Do the following to enable Hyper-V on VMware ESXi:

• Add hypervisor.cpuid.v0 = FALSE to the VM configuration

• Add —-:—-:—-:—-:—-:—-:–h-:—- to the CPU mask for Level 1 ECX (Intel)

For another take, check out William Lam’s post on this topic.

Parting Thoughts

Given the right hardware, it is possible to create a fully-functional VMware test lab that is completely virtual. Go ahead and experiment with the Distributed Virtual Switch, vShield, vCloud Director, and everything else without deploying a ton of servers and storage.

How are you taking advantage of a virtual vSphere environment?

Related posts:

1. VMware ESX 4 can even virtualize itself
2. New VMware ESXi Management Kit
3. Taking snapshots of VMware ESX 4 running in a VM
4. PowerShell Prevents Datastore Emergencies

What to do in a heterogeneous environment?  This can be a challenge for PXE because it requires DHCP — it is difficult to have more than one PXE server on a network. While it is possible to configure a WDS server to install non-Windows operating systems, NFS support can be an issue.  Besides that, do you really want to install Linux via IIS?

Deploying a Linux system configured with DHCP, TFTP, HTTP, and NFS on a separate VLAN is certainly one approach.  But wouldn’t it be convenient if the WDS and Linux PXE servers could coexist peacefully on the same network?

There is a way…

As it turns out, thanks to the lesser-known pxechain utility, it is possible to seamlessly jump from one PXE host to another.  With a few tweaks to your WDS server, you can continue to use it for Windows OS installs and bounce over to a Linux host for Linux, ESXi, or rescue-CD purposes.

Windows Deployment Server Modifications

To make this possible, alter WDS to serve up a PXELinux menu with options to either proceed with WDS or jump over to a Linux PXE server:

• Download Syslinux 3.86 and extract to a temporary location
• Copy the following three files directly to your WDS x64 boot directory, e.g., D:\RemoteInstall\Boot\x64\
  • core\pxelinux.0
  • modules\pxechain.com
  • com32\menu\menu.c32
• Make duplicate copies of these existing WDS files (should already be present in directory above); they need to have “zero” as the extension
  • pxeboot.n12 -> pxeboot.0
  • abortpxe.com -> abortpxe.0
• Create a directory in x64 named “pxelinux.cfg”
• Create a new text file: x64\pxelinux.cfg\default with the following as a guide:

DEFAULT menu.c32
MENU TITLE WDS PXE Server

LABEL wds
MENU DEFAULT
MENU LABEL Windows Deployment Services
KERNEL pxeboot.0

LABEL abort
MENU LABEL Abort PXE
Kernel	abortpxe.0

LABEL linuxpxe
  MENU LABEL Linux PXE server...
  KERNEL pxechain.com
  APPEND 192.168.4.33::pxelinux.0
  #IP address above is Linux PXE host

To activate, run these two commands from a command prompt on the WDS server:

wdsutil /set-server /bootprogram:boot\x64\pxelinux.0 /architecture:x64
wdsutil /set-server /N12bootprogram:boot\x64\pxelinux.0 /architecture:x64

Action!

Boot a machine from the network and you should get a PXELinux menu that offers a choice:

One other note: the Linux PXE server doesn’t actually need to be on the same network, it just needs to be reachable from the client.

What’s your preferred PXE deployment scenario?

Related posts:

1. VMware ESXi 5 Interactive PXE Installation Improvements
2. Which guest operating systems can be customized by SCVMM?
3. Installing vCenter on Linux Technical Preview
4. Getting eth0 back in a SLES for VMware clone

However, one limitation that some have discovered while using this lab-testing technique is the lack of ability to use snapshots with virtual ESX systems.  In fact, after taking a snapshot of a virtual ESX VM, you will see the system boot into the recovery shell like so:

Ouch.

Can we fix it?

I asked some super-smart engineers inside VMware about the issue, and of course they know all about it and how to make it work.

The solution is to enable an undocumented advanced configuration option on the physical ESX host like so:

In other words:

• Log into the console of the physical VMware ESX 4 host
• Run the following command: esxcfg-advcfg -s 1  /COW/PermitVmfsOnRedoHierarchy
• Verify the setting with the “get” flag: esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy

Do I even need to say that this stuff is not supported by VMware Global Support Services (GSS)?  Well, I just did.

Yes we can!

After executing that one command on your physical machine, take a snapshot and boot up your virtual ESX:

Everything works just like you need it to now — you can even create a snapshot hierarchy:

Now what?

Were you constrained by the lack of virtual ESX snapshots?  What interesting things will you use this for?

Related posts:

1. Half the VM reboots on Patch Tuesday
2. VMware vSphere 4 has a Snapshot Alarm
3. Hyper-V snapshots: not for production
4. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off

If you are not familiar with CapacityIQ (a.k.a. CapIQ or CIQ), it is an add-on management product for vCenter Server that assists in planning and managing virtualization resources.

CapIQ is very easy to deploy since it ships as a virtual appliance in OVF format — just load it onto an ESX host and perform a few simple configuration steps. It’s also simple to use since the user interface is a vSphere Client plug-in.  That means no additional web interfaces or user accounts to deal with — just a single pane of glass.

To learn more, check out the Evaluator’s Guide and other documentation.

Installation Experience

For those curious about the installation experience, here is an overview of the very simple procedure.

Upload the CapacityIQ VM using vSphere Client

Configure Credentials

The CapacityIQ appliance has two user accounts: root and ciqadmin.  When you first power on the VM, you will be prompted to enter passwords for those accounts.  Unlike some VM appliances, there are no default passwords involved.

Connect CapacityIQ to vCenter Server

After the appliance is finished booting, a URL will be shown on the console.  Open that with your browser and log in using the ciqadmin account.

Enter the address and credentials needed to connect to your vCenter Server:

vSphere Client Interface

After CapacityIQ is registered with your vCenter Server, restart your vSphere Client.  Verify that the plug-in has been installed automatically.

If everything is in order, you’ll find a new icon on the vSphere Client Home screen:

Views and Reports

CapacityIQ comes with a number of preconfigured views and reports, as seen here:

Related posts:

1. VMware vCenter Server 4.0 installation video
2. VMware vCenter Server Appliance and vRAM Reporting
3. Installing vCenter on Linux Technical Preview
4. Red Hat Enterprise Linux is not Enterprise Virtualization

NEW: VMware vSphere 5 makes this even easier and supports nested 64-bit guests.

Running VMware ESX inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. It is pretty common to do this on VMware Workstation nowadays — the first public documentation of this process that I know of was published by Xtravirt a couple of years ago.

But what if you prefer to run ESX on ESX instead of Workstation?

You may be pleased to know that the GA build of ESX 4 allows installing ESX 4 as a virtual machine as well as powering on nested virtual machines — VMs running on the virtual ESX host.  You can even VMotion a running virtual machine from the physical ESX to a virtual ESX — on the same physical server!

VMware vSphere 4.1 UPDATE: VMware ESXi 4.1 has a keyboard issue when virtualized on an ESX 4.0 host.  In order to virtualize ESXi 4.1, the underlying host must be 4.1.  However, ESX 4.1 classic will work on ESX 4.0.

The extra tweaks to make it all work are minimal, and I will show you how without even opening up a text editor.

After installing ESX 4 onto your real hardware, configure as desired and enable promiscuous mode on a vSwitch:

Create a new VM with the following guidance (choose “Custom”):

• Virtual Machine Version 7
• Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
• 2 VCPUs, 2GB RAM
• 2 NICs – e1000
• LSI Logic Parallel
• New disk – reasonable size greater than 10GB for ESX classic

After you have the VM ready, simply attach a VMware ESX 4 ISO image, power on, and install ESX as a guest OS.

After installation, add the new virtual ESX to vCenter 4 and create a new VM.

If you do not need to run VMs on your virtual ESX, you can stop there.  However, if you try to power on that nested VM, you will see the following error:

To prevent this, just one tweak is needed:

• Shut down the virtual ESX VM
• Click Edit Settings
• Click the Options tab
• Click Advanced / General / Configuration Parameters…
• Click Add Row
• For the Name/Value enter: monitor_control.restrict_backdoor / TRUE

The above procedure is just an alternative to hand-editing the .vmx file — if you prefer to do it that way, feel free.

Now you are ready to power your virtual ESX VM back on, as well as the nested VMs.  This capability should come in handy as you start investigating the new features of vSphere 4.

UPDATE:  In order to take snapshots of a VM running ESX, a quick configuration change is necessary on the physical host .

You may be interested in this related post where a VM is migrated between the physical and virtual ESX hosts.

Related posts:

1. VMware vSphere can virtualize itself + 64-bit nested guests
2. VMotion from physical ESX 4 to virtual ESX 4
3. Taking snapshots of VMware ESX 4 running in a VM
4. If VMware ESXi 4 is so small, why is it so big?