VCritical » Technical

Taking snapshots of VMware ESX 4 running in a VM

Eric Gray — Fri, 26 Feb 2010 19:08:38 +0000

Clearly, the capability introduced with VMware vSphere 4 that allows VMware ESX 4 to virtualize itself is a real crowd-pleaser.

However, one limitation that some have discovered while using this lab-testing technique is the lack of ability to use snapshots with virtual ESX systems. In fact, after taking a snapshot of a virtual ESX VM, you will see the system boot into the recovery shell like so:

Ouch.

Can we fix it?

I asked some super-smart engineers inside VMware about the issue, and of course they know all about it and how to make it work.

The solution is to enable an undocumented advanced configuration option on the physical ESX host like so:

In other words:

Log into the console of the physical VMware ESX 4 host
Run the following command: esxcfg-advcfg -s 1 /COW/PermitVmfsOnRedoHierarchy
Verify the setting with the “get” flag: esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy

Do I even need to say that this stuff is not supported by VMware Global Support Services (GSS)? Well, I just did.

Yes we can!

After executing that one command on your physical machine, take a snapshot and boot up your virtual ESX:

Everything works just like you need it to now — you can even create a snapshot hierarchy:

Now what?

Were you constrained by the lack of virtual ESX snapshots? What interesting things will you use this for?

VMware vCenter CapacityIQ 1.0.1 for vSphere

Eric Gray — Tue, 15 Dec 2009 18:18:52 +0000

Today VMware released vCenter CapacityIQ 1.0.1 — an update to support VMware vSphere 4. Like vSphere, CapacityIQ is offered with a 60-day trial.

If you are not familiar with CapacityIQ (a.k.a. CapIQ or CIQ), it is an add-on management product for vCenter Server that assists in planning and managing virtualization resources.

CapIQ is very easy to deploy since it ships as a virtual appliance in OVF format — just load it onto an ESX host and perform a few simple configuration steps. It’s also simple to use since the user interface is a vSphere Client plug-in. That means no additional web interfaces or user accounts to deal with — just a single pane of glass.

To learn more, check out the Evaluator’s Guide and other documentation.

Installation Experience

For those curious about the installation experience, here is an overview of the very simple procedure.

Upload the CapacityIQ VM using vSphere Client

Configure Credentials

The CapacityIQ appliance has two user accounts: root and ciqadmin. When you first power on the VM, you will be prompted to enter passwords for those accounts. Unlike some VM appliances, there are no default passwords involved.

Connect CapacityIQ to vCenter Server

After the appliance is finished booting, a URL will be shown on the console. Open that with your browser and log in using the ciqadmin account.

Enter the address and credentials needed to connect to your vCenter Server:

vSphere Client Interface

After CapacityIQ is registered with your vCenter Server, restart your vSphere Client. Verify that the plug-in has been installed automatically.

If everything is in order, you’ll find a new icon on the vSphere Client Home screen:

Views and Reports

CapacityIQ comes with a number of preconfigured views and reports, as seen here:

Easy recovery from a full VMware ESX datastore

Eric Gray — Tue, 06 Oct 2009 21:57:37 +0000

This is the third article in a series on VMware vSphere thin-provisioned virtual disks. Now that we’ve covered:

You may be nearly convinced to start using thin provisioning, but still wondering…

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically grow to accommodate additional storage demand. When VMware ESX detects this condition, virtual machines in need of additional storage are instantly paused to prevent guest operating systems from failing. Conversely, VMs that that read and write to existing allocated storage blocks will continue running without issue — not all virtual machines will be paused just because a datastore is out of space.

If you ever find yourself in this situation, it’s not hard to fix. Here is one simple approach, step-by-step:

Free up some space by deleting or moving files — ISO images or powered-off VMs would be perfect
Resume one of the paused VMs
Use Storage VMotion to move the disks for that VM to another datastore
Resume the remaining VMs

Watch the procedure in action:

Depending on the size and storage demand of each VM, additional migrations may be needed. An alternative resolution would be to add additional space to the SAN LUN and grow the VMFS volume.

The Experiment

To simulate a sudden storage demand by the thin-provisioned VMs in the above video, I simply copied a large file from a network share to each Windows Server 2003 VM simultaneously.

For the curious, below is a PowerShell script for the task. Run it from anywhere — it uses Sysinternals psexec to remotely initiate a file copy on each VM from a network share.

# VM naming convention combines this string with 2-digit number
$vmPrefix = "VM2003e-"
 
# UNC path to a large file that will be copied into each VM
$iso="\\fileserver\ISO\large.iso"
 
# credentials to download above file from \\fileserver
$user="domain\username"
$pass="pa55word"
 
# pass this function a list of numbers
function createVmList ($series) {
    $vmList = @()
 
    foreach ($id in $series) {
        $vmList +=  %{"$vmPrefix{0:00}" -f $id}
    }
    $vmList
}
 
# make sure Sysinternals psexec is in your path
function psexecOnVm ($cmd, $vmList) {
    $vmList | % {Invoke-Expression "psexec.exe \\$_ -d -u $user -p $pass $cmd"}
}
 
psexecOnVM "cmd /c copy /Y $iso c:\" (createVMlist (31..39))

VMware ESX is Resilient

You may have been surprised at how easy it is to recover from a full datastore — without so much as a guest OS reboot. It’s a testament to the rock-solid architecture behind VMware ESX and VMFS. No other virtualization platform comes close. Try for yourself. See what happens if a group of thin-provisioned Hyper-V virtual machines suddenly run out of storage — it’s not going to be pretty.

PowerShell Prevents Datastore Emergencies

Eric Gray — Fri, 02 Oct 2009 10:30:03 +0000

In my previous post on VMware vSphere thin provisioning, I pointed out the new datastore alarm feature. You can take advantage of this feature to respond to a sudden storage demand and automatically take action before end users notice.

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script. Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore. VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users. Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
 
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
 
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
 
if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished. As a workaround, it can be launched from cmd.exe as long as it receives something on standard input. To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command. You also need to decide whether to run one time or repeat the action.

Action!

To test the alarm, either fill up the datastore or temporarily lower the alarm threshold. When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Note the “Initiated by” column — that’s the machine account for this vCenter Server. The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore. Here you can see a sample sequence of events, newest on top:

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore. It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion. Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

Use Coreinfo to view VM core and socket count

Eric Gray — Wed, 23 Sep 2009 03:58:19 +0000

A VMware ESX 4 (and ESXi 4) virtual machine can have up to 8 virtual CPUs. Even though the underlying physical hardware has multiple cores per socket, each virtual CPU presented to the guest OS appears as a separate socket.

This normally does not matter because guest operating systems treat cores as CPUs anyway. However, there is one significant area that is affected — licensing. Some operating systems and applications are licensed by physical CPU, regardless of the core count. If this license is enforced only by a legal agreement, and not by technology, then there is still no problem — configure a four vCPU VM on a system with a quad-core processor and you are in the clear.

The issue arises when software enforces the CPU socket limitation. One good example of this is an operating system like Windows XP, which supports a maximum of two sockets — there is no way to use four cores if each is presented as a socket even though such an action would still be allowed within the license agreement.

To address this, vSphere 4 introduced a new undocumented VM configuration option called cpuid.coresPerSocket.

If you have experimented with the feature — Duncan Epping already wrote about it a while back — you may have wondered if there is a method to see how many cores and sockets a Windows system has. After all, Device Manager simply shows each core as a CPU.

One of the more recent Sysinternals tools from Mark Russinovich does that very thing.

Acquiring Sysinternals, and gaining Mark Russinovich, was an exceptionally wise move by Microsoft. So far, so good — knock on wood — Microsoft has not done anything ridiculous like restricting the tools to be part of a Software Assurance (SA) incentive.

The Sysinternals utility is called Coreinfo. In these examples I am using it with two flags to show cores and sockets:

coreinfo -c -s

Here is a default VMware ESX 4 VM with 8 vCPUs:

After changing cpuid.coresPerSocket to 8, like so:

The guest OS sees a single socket, 8-core VM:

Setting the value to 4 divides the 8 cores into two sockets like this:

That last setting is the most legitimate for licensing purposes, as the underlying hardware here is a dual quad-core.

Please remember that this feature is currently not supported by VMware GSS. If you need to use it in production, please contact your VMware account team so that your use case can be taken into account when planning future releases.

Leave a Comment • Subscribe to RSS • Subscribe via Email • Follow Eric Gray on Twitter
More articles on: Sysinternals, Technical, VM hardware, vSphere • Browse All Virtualization Content
Use Coreinfo to view VM core and socket count by Eric Gray © 2009 • VCritical

VMware ESX 4 can even virtualize itself

Eric Gray — Tue, 19 May 2009 03:25:45 +0000

Running VMware ESX inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. It is pretty common to do this on VMware Workstation nowadays — the first public documentation of this process that I know of was published by Xtravirt a couple of years ago.

But what if you prefer to run ESX on ESX instead of Workstation?

You may be pleased to know that the GA build of ESX 4 allows installing ESX 4 as a virtual machine as well as powering on nested virtual machines — VMs running on the virtual ESX host. You can even VMotion a running virtual machine from the physical ESX to a virtual ESX — on the same physical server!

The extra tweaks to make it all work are minimal, and I will show you how without even opening up a text editor.

After installing ESX 4 onto your real hardware, configure as desired and enable promiscuous mode on a vSwitch:

Create a new VM with the following guidance (choose “Custom”):

Virtual Machine Version 7
Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
2 VCPUs, 2GB RAM
2 NICs – e1000
LSI Logic Parallel
New disk – reasonable size

After you have the VM ready, simply attach a VMware ESX 4 ISO image, power on, and install ESX as a guest OS.

After installation, add the new virtual ESX to vCenter 4 and create a new VM.

If you do not need to run VMs on your virtual ESX, you can stop there. However, if you try to power on that nested VM, you will see the following error:

To prevent this, just one tweak is needed:

Shut down the virtual ESX VM
Click Edit Settings
Click the Options tab
Click Advanced / General / Configuration Parameters…
Click Add Row
For the Name/Value enter: monitor_control.restrict_backdoor / TRUE

The above procedure is just an alternative to hand-editing the .vmx file — if you prefer to do it that way, feel free.

Now you are ready to power your virtual ESX VM back on, as well as the nested VMs. This capability should come in handy as you start investigating the new features of vSphere 4.

You may be interested in this related post where a VM is migrated between the physical and virtual ESX hosts.

VMware vCenter Server performance stats levels

Eric Gray — Mon, 20 Apr 2009 17:43:54 +0000

In a recent post, I explained the essential role that your database job scheduler (e.g., SQL Server Agent) plays in keeping the vCenter (VirtualCenter) database size under control. The statistics level is another critical factor that affects your database storage and performance requirements.

The statistics levels are easily — perhaps too easily — configured by using the VI Client:

You can read in more depth about the performance statistics in the Basic System Administration manual and the VirtualCenter Monitoring and Performance Statistics Tech Note, which has this to say about stats level selection:

Level 1 has very low overhead on both the VirtualCenter server as well as the ESX Server hosts. Levels 2 – 4 have slightly greater overhead on ESX, but can adversely impact VirtualCenter performance if there are more than 10 ESX Server hosts. In particular Level 4 can quickly fill the VirtualCenter database, so it should only be turned on for limited periods of time.

When it says Level 4 can quickly fill the VirtualCenter database, that’s not hyperbole. Take a look at this chart to see what happens when the “5 minute” interval is changed from level 1 to level 4:

The VPX_HIST_STAT1 table — the one that corresponds with the “5 minute” interval — immediately begins consuming many times its previous size. After a day, the number of rows in that table increased about 20 times! (The tick marks across the x-axis of this chart each represent an hour.)

Recovery is simple, but not immediate. After resetting to level 1, the additional rows disappear through attrition as the rollup process does its job.

The statistics levels are configurable in order to collect more detailed performance data on your VMware infrastructure. As with most things, there is a tradeoff — making it critical for administrators to know that a seemingly simple change via the UI can put a huge hit on your database — not only in storage requirements but performance, too. Do yourself a favor and talk to your DBA before changing your vCenter Server performance stats levels.

VMware vCenter really needs an agent

Eric Gray — Fri, 17 Apr 2009 02:24:26 +0000

One of the great features of VMware vCenter Server (a.k.a. VirtualCenter, or VC) is the performance charts for virtual machines and other objects in the inventory. Here is a typical chart:

Managing the data behind these charts is actually a very demanding responsibility of vCenter: receiving, storing, and later aggregating to keep the database size under control. Starting with VC 2.5, a couple of big changes were made in this area to help with scalability: the schema was modified to divide stats data into multiple tables, and the rollup duty (data aggregation) was delegated to the database server’s native job scheduling mechanism. This article focuses on SQL Server and the SQL Server Agent; there is an equivalent capability in Oracle as you would suspect.

The VC 2.5 installer creates a set of three jobs (for past day, month, and week) and throws up a notice reminding you to make sure the SQL Server Agent is running on your database server — for good reason, ignore it at your own peril. You can see the jobs in the SQL Server Management Studio.

You’ll notice in this example there are actually three instances of each of the three jobs because there are multiple vCenter databases hosted on this SQL Server. While not typical, it makes perfect sense in a test environment.

Backstory In the initial VC 2.5 release, the jobs always had a static name, thus making it impossible to run multiple vCenter Servers against one SQL Server instance. I knew that was going to annoy a lot of people, so I filed a bug to have it changed — finally got it in 2.5 Update 2.

What happens if the SQL Server Agent is stopped and these rollup jobs quit running? A picture is worth a thousand words:

In this chart, the abrupt increase in database rows corresponds to the point in time that the rollup jobs were disabled. vCenter continues receiving performance stats from hosts and dutifully depositing them in VPX_HIST_STAT1.

Essentially, the most granular data (VPX_HIST_STAT1) starts piling up, never aggregating into the summarized tables for longer-term storage. In a large environment, it is not uncommon to see millions of rows in these tables. If something goes wrong with the rollup jobs, those millions start multiplying in a hurry.

One nice thing about having the rollups controlled by the SQL Agent is that VC does not even need to be running for the process to take place, which also takes some load off of the VC server. After resuming the jobs, the data flood quickly subsides and things go back on track:

One note about SQL Server Express, the bundled database included with VC: There is no SQL Agent in that product. Before you panic, please realize that VMware engineers know this. For those deployments, the rollup stored procedures are simply kicked off by vCenter Server (vpxd), just like VC 2.0.x. I could not believe the creative workarounds some well-meaning folks have attempted to implement because of this situation — it had something to do with the Windows task scheduler. Please do not do that!

Hopefully now you have a little better understanding of the relationship between your database job scheduler, vCenter Server, and the size of your vCenter database.

VM Encapsulation

Eric Gray — Tue, 17 Feb 2009 23:47:34 +0000

Encapsulation is one of the four key benefits of VMware virtual machines. On the surface it may seem like this is a common feature across all virtualization platforms — but it’s not.

If you are a VMware ESX administrator, you know that a VMware virtual machine consists of several files, normally contained in a single directory. There is a configuration file, virtual disk files, and a few other supporting files. Having all necessary VM files in one directory is the essence of encapsulation.

Let’s say that you have a development environment — not protected by VMware HA — that experiences a host failure. You repair the host, reinstall ESX, and reattach the disk that contains your virtual machines. In order to have ESX recognize a VM, it must be registered. This is easily done with the Datastore Browser, just right-click on the .vmx file and select “Add to Inventory.”

This task can also be performed on the ESX service console (i.e., not ESXi) by using the vmware-cmd utility. Here is an easy technique, just cd into the VM directory first:

vmware-cmd -s register $PWD/whatever.vmx

After that you are ready to power on your VMs and get back to work — snapshots and all.

Recovering Hyper-V VMs

Unfortunately, with Hyper-V you need to be more proactive — unless a VM is explicitly “exported” it cannot be “imported.” There is no supported method of importing a VM from a failed host, especially if it has snapshots. Or is that checkpoints?

Don’t believe me?

For the curious, there evidently is an unsupported technique to put Humpty Dumpty a Hyper-V VM back together again — for data recovery purposes only. If making symlinks to XML files is your thing, go check it out.

When choosing a virtualization platform, don’t forget about the little things that can make a big difference.

Leave a Comment • Subscribe to RSS • Subscribe via Email • Follow Eric Gray on Twitter
More articles on: ESX, Hyper-V, PASS, Technical, VI Client, VMware • Browse All Virtualization Content
VM Encapsulation by Eric Gray © 2009 • VCritical

Better console for PowerShell and VITK

Eric Gray — Thu, 05 Feb 2009 11:00:37 +0000

The stock Windows command prompt (cmd.exe) is pretty limited–if you want to use a better font or copy text in lines instead of clunky rectangular blocks, you need something else. For Windows command-line utilities, PowerShell, and the VI Toolkit (for Windows), I have been using the open-source Console2 with much satisfaction. The tabbed interface is pretty cool:

If that looks interesting, here is what you do:

Download Console2 (current version is 2.00 Beta, build 142); extract on your system somewhere
Grab this file with icons and a replacement configuration file (console.xml); extract into the same location
Make a shortcut to console.exe
- Multiple tabs can be opened during launch by adding -t to your shortcut target (e.g., -t Console2 -t “VI Toolkit”)
Optional: fix your prompts

My configuration file has a few changes to the mouse behavior:

Left – select + copy automatically
Right – paste
Middle – pop-up menu

If you prefer to not use my entire configuration file, just copy the applicable portions from the element.

I hope you have enjoyed this slight diversion from the regular VCritical programming lineup. Thanks for reading.