Taking snapshots of VMware ESX 4 running in a VM

Clearly, the capability introduced with VMware vSphere 4 that allows VMware ESX 4 to virtualize itself is a real crowd-pleaser.

However, one limitation that some have discovered while using this lab-testing technique is the lack of ability to use snapshots with virtual ESX systems.  In fact, after taking a snapshot of a virtual ESX VM, you will see the system boot into the recovery shell like so:

Ouch.

Can we fix it?

I asked some super-smart engineers inside VMware about the issue, and of course they know all about it and how to make it work.

The solution is to enable an undocumented advanced configuration option on the physical ESX host like so:

In other words:

  • Log into the console of the physical VMware ESX 4 host
  • Run the following command: esxcfg-advcfg -s 1  /COW/PermitVmfsOnRedoHierarchy
  • Verify the setting with the “get” flag: esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy

Do I even need to say that this stuff is not supported by VMware Global Support Services (GSS)?  Well, I just did.

Yes we can!

After executing that one command on your physical machine, take a snapshot and boot up your virtual ESX:

Everything works just like you need it to now — you can even create a snapshot hierarchy:

Now what?

Were you constrained by the lack of virtual ESX snapshots?  What interesting things will you use this for?

Related posts:

  1. Half the VM reboots on Patch Tuesday
  2. Hyper-V snapshots: not for production
  3. VMware vSphere 4 has a Snapshot Alarm
  4. IGT Part 5: Hyper-V snapshots are not gone until the VM is powered off
  5. VMware ESX 4 can even virtualize itself

Tags: , , , , ,

  1. NiTRo’s avatar

    I guess ESXi is also concerned ?

    Reply

    1. Eric Gray’s avatar

      Hmm. I have not taken a look at ESXi snapshots in a VM. I suspect that since ESXi does not have the same VMFS architecture as ESX, it might just work as is.

      Reply

  2. Rick Vanover’s avatar

    This is good to know. But this also brings up a big question on how we use ESX as a VM. When I do it, I don’t use the local datastore of the guest ESX host. I always point to a defined iSCSI resource that is constant, regardless of the host state. Further I maintain snapshots of the guests outside of a virtualized ESX host state.

    Maybe I go about it differently, but this lets me keep images and VMs constant in spite of what host hopping and chopping I do.

    I, too am interested on how this applies to ESXi – as most of my nested virtualization does use ESXi.

    Thanks as always for another good post, Eric.

    Reply

  3. William Lam’s avatar

    As Eric mentioned, the advanced host configuration does not apply to ESXi.

    I’ve tried this in the past and saw some interesting results when snapshotting a vESXi host while it was offline.

    I just tried on a powered on ESXi host and basically took a snapshot while it was on, then created a dummy vSwitch and renamed the datastore. I then reverted the snapshot and only the vSwitch was removed, the renamed datastore persisted even after a revert. I need to do more testing but I think with ESXi, it might not work 100%.

    Keep you guys posted when I can test further with both online/offline snapshots for vESXi

    —William

    Reply

  4. Vladan’s avatar

    I just wanted to share my experience. But with little difference… I’m using Workstation 7 and virtual ESXi as a VM. Snapshots works fine. I keep my snapshot files in the same directory of my individual linked clones.

    I did a snapshot test with Powered On vESXi 4, and then revert back.. everything worked. Even password change between those 2 snapshots.

    Thanks
    Vladan

    Reply

  5. Rick Vanover’s avatar

    @Vladan: Does Workstation let you put x64 VMs as a guest on the vESXi server? Or does it have the same hurdle as ESX in this regard.

    Reply

    1. Vladan’s avatar

      Rick,

      you’re right. Just verified. If I want to put x64 VM on the virtual ESXi 4, there is a message saying that the destination does not support 64bit hosts. (fyi. tried to use vmware converter to “send” the VM there, not a direct install of x64 guest…)

      The same trouble on the W7.

      Vladan

      Reply

  6. Rick Vanover’s avatar

    @Vladan Cool – Thanks for verifying. Saved me some time.

    Reply

  7. David’s avatar

    Eric,
    not a question about snapshots or ESXi, but what applications do you use that creates the torn edge on the images?
    David

    Reply

    1. William Lam’s avatar

      It looks like snag-it, I know it can do tears and other type of borders

      –William

      Reply

  8. David’s avatar

    Thanks William & Eric

    Reply

  9. Charles Killmer’s avatar

    This also affects virtual hosts that are moved with storage vmotion. This makes sense as the svmotion process uses snapshots, though I find it curious as I moved the VM when it was powered down.

    I ended up just reinstalling ESX as I want to keep our hosts in line with support.

    Reply

    1. Charles Killmer’s avatar

      Actually, it looks like it is the simple fact that the new data store is NFS instead of block iSCSI. I was unable to reinstall ESX on the NFS VMDK, but I was able to reinstall by creating a new VMDK on iSCSI. Moving the original failed virtual ESX host back to iSCSI resulted in an inability to boot.

      Reply

  10. Artie’s avatar

    Hi,

    [~]# cat /etc/vmware-release
    VMware ESX 4.1 (Kandinsky)
    [~]# esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy
    Exception occured: Unable to find option PermitVmfsOnRedoHierarchy

    Is this feature named differently in ESX 4.1 (VMKernel Release Build 235786) ?

    thx,
    az

    Reply

    1. Miguel’s avatar

      hello
      I was having trouble running a esx 4.0 on a esxi 4.1, it worked previously, but after doing a cold migration to another host it did not work anymore.
      I’ve found that if you disable ctk ( ctkEnabled=false for both the vm and the scsi devices) going to advanced properties on the vm hosting esx it does work again.
      hope that helps

      Reply

      1. Alfredo’s avatar

        Hi Miguel,

        thanks a lot for sharing your experience, I confirm that I was able to resolve the boot problem simply changing the ctkEnabled=false for both the vm and the scsi devices, without any action on the physical ESX host.

        Reply

  11. Greg Gooch’s avatar

    Is there a way to get this working on ESXi 5?

    Reply

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>