VMware ESX 4 can even virtualize itself

ESX on ESX

NEW: VMware vSphere 5 makes this even easier and supports nested 64-bit guests.

Running VMware ESX inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. It is pretty common to do this on VMware Workstation nowadays — the first public documentation of this process that I know of was published by Xtravirt a couple of years ago.

But what if you prefer to run ESX on ESX instead of Workstation?

You may be pleased to know that the GA build of ESX 4 allows installing ESX 4 as a virtual machine as well as powering on nested virtual machines — VMs running on the virtual ESX host.  You can even VMotion a running virtual machine from the physical ESX to a virtual ESX — on the same physical server!

VMware vSphere 4.1 UPDATE: VMware ESXi 4.1 has a keyboard issue when virtualized on an ESX 4.0 host.  In order to virtualize ESXi 4.1, the underlying host must be 4.1.  However, ESX 4.1 classic will work on ESX 4.0.

The extra tweaks to make it all work are minimal, and I will show you how without even opening up a text editor.

After installing ESX 4 onto your real hardware, configure as desired and enable promiscuous mode on a vSwitch:

vswitch0 promisc

Create a new VM with the following guidance (choose “Custom”):

  • Virtual Machine Version 7
  • Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
  • 2 VCPUs, 2GB RAM
  • 2 NICs – e1000
  • LSI Logic Parallel
  • New disk – reasonable size greater than 10GB for ESX classic

After you have the VM ready, simply attach a VMware ESX 4 ISO image, power on, and install ESX as a guest OS.

Virtual ESX 4

After installation, add the new virtual ESX to vCenter 4 and create a new VM.

vSphere Client with virtual ESX

If you do not need to run VMs on your virtual ESX, you can stop there.  However, if you try to power on that nested VM, you will see the following error:

You maynot power on a virtual machine inside a VM.

To prevent this, just one tweak is needed:

  • Shut down the virtual ESX VM
  • Click Edit Settings
  • Click the Options tab
  • Click Advanced / General / Configuration Parameters…
  • Click Add Row
  • For the Name/Value enter: monitor_control.restrict_backdoor / TRUE

Editing VM configuration.

The above procedure is just an alternative to hand-editing the .vmx file — if you prefer to do it that way, feel free.

Now you are ready to power your virtual ESX VM back on, as well as the nested VMs.  This capability should come in handy as you start investigating the new features of vSphere 4.

UPDATE:  In order to take snapshots of a VM running ESX, a quick configuration change is necessary on the physical host .

You may be interested in this related post where a VM is migrated between the physical and virtual ESX hosts.

Related posts:

  1. VMware vSphere can virtualize itself + 64-bit nested guests
  2. VMotion from physical ESX 4 to virtual ESX 4
  3. Taking snapshots of VMware ESX 4 running in a VM
  4. Pentacore CPUs in VMware ESX 4
  5. If VMware ESXi 4 is so small, why is it so big?

Tags: , , ,

  1. Rick Vanover’s avatar

    Eric: Good stuff. I think this is a great way for administrators to get started with the base functionality of the product without allocating a tremendous amount of hardware. This won’t help too much with upgrades, but will definitely help with the base product!

    Cheers.

    Reply

    1. PAramesh’s avatar

      Thx you very much . it was very use full for me .. i don’t find any Youtube video on this ..

      Reply

      1. Louis Ndjouou’s avatar

        This was very helpful stuff especially for testing environment. I actually had two ESXi 4.0 running as Guess within ESXi 4.0 on the physical machine.

        Reply

    2. Mike’s avatar

      Thanks, this works well. I do have a problem loading the tools on the vm’s I create on the virtual esx. Does anyone have any ideas.

      Thanks

      Reply

      1. Mike’s avatar

        i’ve also noticed that you can’t clone a vm from one virtual esx to another virtual esx.

        Reply

        1. Eric Gray’s avatar

          You should be able to come up with a way to do this, but there are some issues — such as duplicate MAC addresses — that need to be dealt with.

          Feel free to chime in if anyone out there has some tips for Mike.

          Reply

  2. Hany Michael’s avatar

    Eric,

    Great post. I’ve been struggling to achieve this since long time with no luck. Unfortunately though its not working on the ESX 4.0 RC, so I guess I’ll have to wait for the GA. Besides, I believe you need to add this inside the nested VMs config: mx.allowNested = “TRUE” ? or it’s not required any more?

    thanks

    Reply

  3. Eric Gray’s avatar

    Rick, you are probably right about not helping with upgrades from ESX 3.x, but going forward this environment could be used to test the ESX 4 patches and upgrades. Actually, I’m told that ESX 3 will also run in a VM like this but it is hardware dependent, and I have not tried it myself.

    Hany, that additional configuration element is no longer needed — it was in the past for certain configurations. By the way, internally VMware has been running ESX on ESX for a long time. A special build was required, though.

    Anton, spasebo vam! It’s very cool to see my articles translated into Russian. Especially if you take a look at my background on the About page…

    Reply

    1. Mike’s avatar

      Hi Eric,

      I am using esx on esx. I have a number of w2k3 vm’s running under it. I’m having problems loading the tools. It says Vmware tools should be installed inside a virtual machine.

      Any ideas? Is there another switch I need to set.

      Thanks
      Mike

      Reply

  4. Justin’s avatar

    Eric,
    Would you mind testing out installing ESX 3.5 as a VM on ESX4? This would be huge for us here!

    Reply

  5. Mike Laverick’s avatar

    Great post. I would have posted something similiar myself but I was told I would be in breach of the NDA agreement I’d signed…

    Reply

  6. Eric Gray’s avatar

    Justin, you can run ESX 3.5 in a VM on certain hardware — AMD Barcelona and Shanghai, probably Intel Nehalem.

    Reply

  7. Hany Michael’s avatar

    Eric,

    I’m a bit confused here. I know that the ESX VM is 4.0, but what about the underlying physical ESX host, does it have to be ESX 4.0 also, or it can be ESX 3.5?

    I just tired the ESX 4.0 GA as a VM on ESX 3.5U3 but it didn’t work for me, the VM starts then a I get a blank screen and nothing happens, few seconds later the VM shutdown by itself and no errors are given.

    Any ideas?

    Thanks

    Reply

    1. Eric Gray’s avatar

      In my example, the physical host is also running ESX 4.

      Reply

  8. Yue Yao’s avatar

    Hi Michael,
    In my testinstallation, vESX4.0(GA,RC1) is running on physical ESX3.5U4(but I have trouble with vswif0 after installation), and on W2K3+VMWorkstation 6.5.2.

    Reply

  9. Astor’s avatar

    Eric,
    I try this with a ESXi4 final, but I can install and boot only Esxi3.5(very unstable).
    I can not install ESXi4, ESX3.5 and ESX4(Final)(PSODs and more PSODs).
    Where I download this version (GA)?
    My hardware is a whitebox(Athlon64 X2 7750 extreme)
    Sorry for my english, thanks.

    Reply

  10. Hany Michael’s avatar

    thanks Yue Yao for the note, in my case the vESX4.0 boots/run fine, and i have no issues whatsoever with the networking, it’s just the VM running within the vESX4.0 that is not work for me, I’m using physical ESX3.5U3, so I’ll upgrade to U4 and see how it goes…

    I just hope Eric won’t kick us out of here :p

    Reply

  11. Eric Gray’s avatar

    Astor: by GA I am referring to the official (final) release — not beta/RC.

    Unfortunately, it’s hard to say why folks are experiencing mixed results without digging in deeper.

    Reply

  12. TimB’s avatar

    To get ESX 4 hosts working as VMs within your ESX 3.5 environment do the following:
    1) Create the VMs to run ESX4 on an ESX 3.5 host.
    2) Add these VMs to the inventory on an ESX4 host.
    3) Build the virtual ESX 4 hosts as above.
    4) Once built, these VMs should run in the ESX3.5 environment.

    Sadly, vmware tools do not work. Any ideas?

    Reply

  13. Rene’s avatar

    Hi,

    Can you start a nested windows x64 VM under the vESX 4.0 (VM) ?

    Reply

    1. Eric Gray’s avatar

      Rene, I do not believe 64-bit nested VMs are possible at the present.

      Reply

  14. rene’s avatar

    Can you verify this for me? Would like to know if window 2008 x64 would run (without hyper-v)

    Reply

  15. Heiko Verlande’s avatar

    rene,

    I can verify x64 VM’s aren’t running within the vESX 4.0 (VM). Only 32 bit hosts boot. More info can be found here. Hope this answers your question.

    Reply

  16. Vladan’s avatar

    I came a bit lat to this post because I was away of my Island at holidays in France. It sounds great all this.

    In the pas Xtravirt published a white paper for ESX 3.0 and 3.5. I could learn a lot from this. Now They did the same thing again… For ESX 4.0.

    http://www.vladan.fr/vsphere-4-in-vmware-workstation/

    Reply

  17. Eric Gray’s avatar

    Heiko, thanks for pointing that out for Rene.

    Welcome back, Vladan.

    Reply

  18. akabdog’s avatar

    Great info, I was able to get 4.0 working in 3.5U4, once I modified the vSwitch.
    If anyone is interested, I got it working with 768 MB of memory (instead of 2GB)- documented here http://infinitedisorder.com/?p=158

    Reply

  19. Eric Gray’s avatar

    Akabdog – cool, thanks for the contribution.

    Reply

  20. Ali’s avatar

    Hi,

    I installed ESXi 4.0 base host & couple of ESXi servers in there as a VM.
    Now I am unable to run nested windows 32 bit VM’s on those ESXi servers.
    All I get is a blank console while booting the nested VMs.
    I even tried deploying a VM from Marketplace, however, the VM gets deployed, gets poweredON, but only a blank console to look at.
    No error messages as such while powering ON the nested VM.

    Thanks in advance.
    Ali

    Reply

  21. Neil’s avatar

    Hi,

    Great article. I’ve been able to setup a complete vSphere test rig on a single €200 ML115.

    One issue I have is that I can’t add hosts to a dvSwitch. I have available NICs but the Add Hosts list is blank. Has anyone had any luck with this?

    Thanks,

    Neil.

    Reply

  22. Neil’s avatar

    Hi,

    I just answered my own question. dvSwitches are only available with the Enterprise Plus and evaluation licenses, so NFR licenses don’t work as they are just Enterprise.

    Neil

    Reply

  23. rahul’s avatar

    eric: I was successfull installing virtual ESX 4.0 inside physical ESX 4.0. But I am not able to power on VM’s inside Virtual ESX. I had did the configuration changes as you have mentioned above. Could you please help me in this?

    Reply

  24. Eric Gray’s avatar

    Rahul, I suspect it is your CPU. What is the exact vendor and model info?

    Reply

  25. Rahul’s avatar

    Eric: I tried it in Dell power edge server and it worked….Thanks a lot…
    keep posting this kinda stuff….its really helpfull.

    Reply

    1. Eric Gray’s avatar

      Good to know, thanks.

      Reply

  26. Shan N’s avatar

    Hi Eric,
    Great Web site and articles. I was going to buy more hardware to build a test environment. Now I am going to try to create another ESX 4.0 server in my physical ESX 4.0. I hope I will be able to test vMotion with this setup.

    Thanks for the great work.

    Regards,
    Shan

    Reply

  27. Mark W’s avatar

    Ali – I am having the same problems with the blank/black console from vSphere (running on the Host. The issue persists in both ESX and ESXi 4.0. I suspect that it may be a Windows 7 issue (host). Like you, I have tried an appliance (2 actually, a Turnkey LAMP app and VA-ntop).
    Next I plan to try:
    Import a Win XP VM and try to RDP to it from within Win7.
    Build a pWinXP, install vSphere and try that.
    Lastly (is that a word?) Install Wkstn on the pXP machine, install vESX and try to get a console from pWin7.

    I will report back, but it might be a week or 2 as I have several pESX boxes to build this week.

    Reply

  28. gogogo5’s avatar

    Hello

    Has anyone been able to successfully view the console of a nested VM regardless of OS. My VMs also have blank/black consoles…

    cheers
    gogogo5

    Reply

  29. rahul’s avatar

    Yes …I had installed WIN2K3 R2 as a VM GOS inside my virtual ESX 4.0 . I didnt find any issue as you are facing. I was able to use my nested WIN2K3 VM console.

    Reply

  30. Sven’s avatar

    Hi Eric,

    please could you talk a bit about your hardware setup and the performance? Are you able to run a cluster of two virtualized ESX to make some VMotion, HA etc.?

    Reply

    1. Eric Gray’s avatar

      Sven,

      You can absolutely exercise most of the features of vSphere with a 2-node virtual ESX cluster. VMotion, HA, Host Profiles, vDS — everything should work the same way. Performance of the nested VMs is not stellar, but you can at least get a feel for all of the functionality. Check out the various posts linked in these comments to see what others are doing with virtual ESX systems.

      Eric

      Reply

  31. Deon’s avatar

    Hi Eric,

    I have 2 x ESX4 hosts running on the same ESX3.5 AMD Dell box. I have configured iSCSI and vmotion on the 2 ESX vm’s. I can successfully see the shared storage on the 2 boxes, and created a vm that is runnign w2k3. Problem i have is that they will not see each other via the service console or vmption network. Do you know of any further network tweaks? I have already set the vswitch promiscuous mode to accept.

    Thanks

    Reply

    1. Eric Gray’s avatar

      Deon, it’s too hard to say for sure. Interesting that you are using ESX 3.5 as the host, did you try with ESX 4?

      Reply

  32. Goris Christophe’s avatar

    Hi,

    Have try this install on a HP DL360 G4, ESX 4 is running fine.

    But when I try to start the virtual ESX4 I receive the following error:

    “This version of ESX requires a CPU capable of 64-bit operation.”

    CPU’s are EM64T as the ESX4 hardware install is working.

    Reply

    1. sripati’s avatar

      Hi Goris,

      I know I am too late in answerin this Question. But I think rest users would be able to get the benefit out of the same.

      Once u create the VM…go to the Option …….check in the option where in you get the BIOS screen at Power on.
      2- Now go the advanced tab in the processor you have 1.4 & 1.1. select 1.1 version….
      press F10..you are ready to go.
      It worked for me(MY White box is Q9550 + S3210 SHLC+ 8 GB).

      Reply

      1. Denesh Ashok’s avatar

        Hi sripati,

        I tried this with my DELL PE 2850 and the host esx is working with ESX 4, and when i followed ur steps above i get the same error still “This version of ESX requires a CPU capable of 64-bit operation.”.

        Then i down graded the esc to 3.5 U5 and still i get the error.

        But I can run ESX4 on the DELL tin still what do i need to do please let me know anyone so I can start working on my lab and get my VCP soon.

        many thanks

        DA

        Reply

  33. Goris Christophe’s avatar

    have run “CPU Identification Utility” from vmware.

    It seems to the CPU’s can not run 64bit as virtual :-(

    Supports 54-bit Longmode: Yes
    Supports 64-bit VMware :No

    Reply

  34. Nicolas’s avatar

    Hi,

    I cannot run VMs on “virtual” ESX4, blank console and 100% CPU,
    is-it related to AMD opteron ? I just bought a poweredge sc1435 for a lab … :(

    Nicolas

    Reply

    1. Chadwick King’s avatar

      I have seen this issue with not enabling Virtualization Technology on the AMD processor in the BIOS – double check and let me know if that works!

      Reply

  35. NiTRo’s avatar

    I’m running 7 ESXi 4 to test HA stuff…
    Thanks again for this good post Eric

    Reply

  36. Guru’s avatar

    Eric,

    I am running VM in ESX 4.0 Evaluation version on my desktop, but unable to power on the VM in it. I just get a black/blank screen and never get any screen even though it says that the VM is powered on. I have an ISO for windows 2003 for which I dont get any installation wizard.

    Guru.

    Reply

    1. Nicolas’s avatar

      Hi Eric

      Do you have an Intel or Amd ?, you MUST have virtualisation instructions enabled ( VT / in the bios).

      Nicolas

      Reply

  37. Guru’s avatar

    Nicolas,

    I have an AMD 64 bit, enabled the bios setting (NX for AMD) for virtualization.

    Guru

    Reply

  38. Nicolas’s avatar

    Ok, on amd there is a special generation level you must have !!, even with VT enabled it does’t work.

    You can search if you have a second generation or a third one this link, you MUST have at least a third one !!

    http://www.cpu-world.com/CPUs/K10/AMD-Third%20Generation%20Opteron%208356%20-%20OS8356WAL4BGD.html

    Reply

    1. Eric Gray’s avatar

      Thanks for your contribution, Nicolas.

      Reply

  39. Guru’s avatar

    I have an Athlon 64.. Not sure about the generation.

    Guru

    Reply

  40. V.Peter.S’s avatar

    Eric, This is great info.
    I had discovered this for 3.5 last year, Check posts from vmware Academy Partners,
    One added issue we have been having with our dell 2950′s is that the install would hang on formatting the vmfs.

    We have been able to resolve this by using a RDM for the disk of the esx VM.

    Pete

    Reply

  41. Rafi’s avatar

    Eric,

    I have a HP DL385 G2 with AMD Opteron Processor. I have enabled the Virtualization in BIOS. This machine has enough RAM and DASD.

    I was able to install ESX 4 on the bare metal machine

    I am trying to get more familar with vSphere and to get familar with other features I needed another ESX server so I followed your post and was able to successfully install a virtual ESX 4i instance within the ESX 4 server. I have successfuly added both the ESX servers in the test cluster in vCenter.

    I have also added a bare metal pentium 4 OpenFiler machine which is working as iSCSI storage. I have successfully configured and openfiler storage is visible to the ESX4 and virtual ESX4i instance

    virtual ESX4i instance allows me to create virtual machines. However when I power them on I am not able to see anything in the console. When I power on the same machine and migrate it to the physical ESX host and I can see the console.

    I am a virtualization newbie who is trying to learn … any assistance will be greatly appreciated.

    Thanks!

    Reply

  42. Fred’s avatar

    Hi Eric… Thx for this post!
    I executed the 2 tweaks and was able to install an ESX3.5 server and an ESX 4 server

    The ESX4 server fails after a reboot. it fails on vsd-mount ….

    Any suggestions?

    The ESX3.5 runs fine

    Thanks
    Fred

    Reply

  43. Fred’s avatar

    It looks like we need another tweak for snapshot usage….. because that was the issue.
    The ESX 3.5 reboots fine (with snapshot in use) but the ESX4 fails on vsd-mount

    Any suggestions?

    Tx
    fred

    Reply

  44. Rick’s avatar

    I’m seen the same issue when I create snapshots on Virtual ESX 4s. I think it has to do something with the fact that the COS is on the same vmfs disk as the one you use for regular vmstorage. I’ve Google and found the issue, but I’m not sure if anybody has found a workaround yet.

    Reply

  45. Steve Shockley’s avatar

    FYI, I had problems getting ESX 4.0u1 to install under ESX 4.0 on an HP DL680c G5. When the install would start formatting the disks, vmkfstools would fail with status 65280, print the vmkfstools command options, then say “Error: not supported”. It turns out I only had the problem when the guest disk was on a volume on a NetApp filer via NFS. Putting the volume on direct-attach disk led to a successful installation.

    Reply

  46. Eric Gray’s avatar

    The snapshot issue may be addressed with an undocumented configuration setting. Please see http://www.vcritical.com/2010/02/taking-snapshots-of-vmware-esx-4-running-in-a-vm/

    Reply

  47. Fred’s avatar

    Thanks for this advanced configuration setting Eric!
    My Physical ESX host is ESX4i and sadly it is an unknown Option in there…

    C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-advcfg.pl –server nl01sol
    vh10 –username root -g /COW/PermitVmfsOnRedoHierarchy
    Enter password:
    Option /COW/PermitVmfsOnRedoHierarchy not found.

    C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-advcfg.pl –server nl01sol
    vh10 –username root -s 1 /COW/PermitVmfsOnRedoHierarchy
    Enter password:
    Option /COW/PermitVmfsOnRedoHierarchy not found.

    Reply

    1. Eric Gray’s avatar

      Since this is an unsupported/undocumented configuration setting, it cannot be controlled via remote API. The good news is that you should be able to just log into the “tech support mode” console of ESXi and run the same command that works on the service console.

      In other words, go to the ESXi console, press ALT+F1, type in the word “unsupported” (blindly, as it is not echoed to the screen), enter your password when prompted, and then run the esxcfg-advcfg command.

      Reply

  48. Fred’s avatar

    Thanks again Eric! i went to “unsupported “service console” on the physical ESX4i and execu ted esxcfg-advcfg -s 1 /COW/PermitVmfsOnRedoHierarchy
    made a snapshot and could startup without any problem.

    Reply

  49. Prashant M M’s avatar

    Hi Eric,

    I have installed Vmware Vsphere on vmware workstation and I am able to create the HA & DRS cluster with FREENAS Shared storage but my problem is when I create the VM and try to power on the VM with the ISO or any other medium, I am getting blank black screen and I am not getting any kind of error but just the blank screen.

    I even changed the points like monitor_control.restrict_backdoor = TRUE etc.

    When I poer on the ESX host VM I am still getting the error that I can’t run the NESTED VMs.

    Reply

    1. Eric Gray’s avatar

      Prashant,

      Could it be that you have an older AMD CPU? If you notice, some of the comments above have had similar issues and one reader felt it was AMD-specific. I have used AMD Shanghai CPUs for this and everything worked properly.

      Eric

      Reply

  50. Tom Galat’s avatar

    Eric,
    Once again — good stuff. I set up ESXi V4 on a Dell R900 W/128GB Ram and 8 CPUs. We have 15 or so test VMs running on it.
    With that said I downloaded the 60 trial of ESX 4.0.u1 and vSphere. Installed a 3 host cluster and turned on HA and DRS. Added 6 guest VMs on the virtualized ESX and all works great, systems are a bit slow but that’s what we get for multilayer hypervisors.
    My question is — has anyone got Fault Tolerance to work? When I turn it on I get a hardware compatibility error. It seems that FT wants VT turned on in the bios, but the VMware bios of the guest ESX server goes not have this option. The backend storage is IB so there is no delay there. Any ideas?
    I need to give a presentation demo to the higher ups and they are interested in FT.

    Reply

  51. tommelise’s avatar

    Hi Eric,

    Thanks for this post!

    I have installed ESX 4 on a white-box (Dell 9200) and it works like a charm.
    Now I’ve followed yours instructions to install ESXi4 as a VM on it.
    And I got a PSOD “No place on disk to dump data”.
    Any suggestions ?

    Thanks

    Reply

  52. Bhaskar’s avatar

    Hi,

    I have two esx servers as guest os on my physical esx host but the two esx guest are unable to communicate each other.
    Please help me in acheiving this feature.

    Thanks
    Bhaskar.

    Reply

    1. Datto’s avatar

      Bhaskar — you need to change the Security on the Service Console of your physical ESX host AND the VM Traffic Port on your physical ESX host to allow Promiscuous Mode (change the setting to Accept instead of the default of Reject). That will then allow communication between your ESX hosts.

      Datto

      Reply

  53. Eric Gray’s avatar

    Hi Eric we have the same name! Anyway, if you need a good company to buy any VMware vSphere 4 products, a good place is my company ITO Solutions. Here is a link to our site for VMware ESX http://www.itosolutions.net/Server_Datacenter_Virtualization_s/523.htm

    Reply

  54. Fabiano Silos’s avatar

    Hi Eric, do you have any clue on how to run 64bit VM inside the ESX that is running over a virtual machine?

    Reply

    1. Eric Gray’s avatar

      Currently not possible…

      Reply

  55. VM_newbie’s avatar

    Hi,
    I have installed ESX 4.0 on workstation 7. the ESX has 22 gig VMFS datastore. I am trying to create a new VM in ESX and dont know how to boot from ISO to install the OS. After creating the VM when I go to connect cd/dvd it did not detect the OS in dvd drive. Do i have to copy the ISO to datastore first if yes then how i can do that or I can only create VM using the VC 4.

    any help appreciated

    Reply

  56. mike’s avatar

    Ok, so I have a problem with my esxi 4 “update 2″ guests running under the same esxi 4 version for the host. The problem is that the guest esxi servers do not have correct time despite having ntp setup just like the host.
    The host esxi server has been pointed to pool.ntp.org and shows the correct time via the vshpere client. The guest/s esxi servers are pointed to the same exact ntp server “pool.ntp.org” but after restarting the service just like I did on the host the time does not change and is off. I manually set the time on the guests to the correct time but after 8-10 hours the time is faster on the guests by about 2 minutes and some change. Are others experiencing time difference on there nested esx/i vm’s? I figured it would use the ntp server and adjust it time correctly just like the host did. I also verified they could resolve and communicated with the ntp server …so that can be taken out of the equation.

    Thanks,

    Mike

    Reply

  57. SargentTi’s avatar

    Great article, help me established a test ENV for Lab manager, thanks!

    Reply

  58. Anantha’s avatar

    Hello Eric

    My Host OS is windows 07, on which I have installed VMware server and ESX 4.0 inside it

    Following is the order

    Win07Ultimate (Host OS)–> VMware-server-2.0.2-203138 –>ESX4.0 –> win 2008 VM

    I tried above solution of setting up the Vswitch in promiscuous mode and adding the line “monitor_control.restrict_backdoor / TRUE”

    But still I get the error message “You may not power on a virtual machine in a virtual machine”

    Is there any solution if the ESX is not directly installed on the hardware as the host OS

    Regards,
    Anantha

    Reply

  59. Onder’s avatar

    Hello all,

    I tried to install the ESX v4 server on ESX v4 server as described in the beginning of this article. The installation cannot even take place.

    Error code:
    CosSup: 406: vmkernel not loaded: cannot PSOD – Basically not supported microcodes.

    My cluster configuration is with EVC AMD Opteron Generation 3. The exact processor type is Ams Opteron Quad-Core 8389.

    Installation of ESX v3.5 is working, but after reboot I have the same error all the time even with different VM settings.

    Error code:
    …….initialization of vmkernel failed, status 0xbad001e – Microcodes problem again.

    Can someone help?

    Thanks.

    Reply

  60. MAZ’s avatar

    Does any one tried to move ESX 4.0 Host VMs to Another ESX4.0 Host.
    I hve Dell 2950 2.6 GHz 8x Core, want to shift my ESX 4.0 Linux VMs from this one to DELL 2950 2.0 GHZ processor.

    Any idea?
    I have VCenter VM also on the same ESX Host1.
    Thanks for advise in advance.

    MAZ

    Reply

    1. Chadwick King’s avatar

      MAZ – You should have no problems moving VM’s. My questions would be is how exactly are you thinking of doing this? Are you wanting to use vMotion? Are these host in a HA/DRS cluster? Answering these questions would help me better understand what answer your looking for.

      Thanks!

      Reply

  61. Chadwick King’s avatar

    Just wanted to note that anyone “experiencing” issues with blank screens and such on AMD platform CPUs be sure to check Vritulization Enabled on the Processor settings in the BIOS. I know this may sound kind of insulting but believe it or not I missed this on my system and yes I felt like a total newbie. I could easily run VM’s in workstations but had random issues with nested VM’s. Just double check and make sure that you weren’t pulling a “chad” and NOT saving the BIOS configuration…. would’ve saved me a headache..

    Reply

  62. anand’s avatar

    Hi,

    I have some problem while installing ESX 4.0 Server in Vmware Work Station 6.5 in Home. Problem is they need any External Network Connection is Failed.

    So Any one of them give me some solutions

    Reply

  63. ram’s avatar

    I cannot create VM’s on Virtual ESX 4.1 server. I followed the notes to create virtual ESX 4.1 on the physical ESX 4.1. When, I try to create VM’s inside virtual ESX 4.1, I get this error –
    host cpu is incompatible with the virtual machine’s requirements at cpuid level register edx

    Any help appreciated.

    Tks / RL

    Reply

    1. NiTRo’s avatar

      you can only do BT in a nested ESX, try 32bit VM ;)

      Reply

  64. Louis Ndjouou’s avatar

    Hello people

    I’m trying to attach an external USB drive on the ESXi host. I can see the USB drive but cannot do much with.

    Is there away to have a USB drive working as an external storage attached to the ESX?

    thanks

    Louis

    Reply

  65. Iain’s avatar

    Just a slight aside here, I have LabManager running inside LabManager at the moment, just thought I’d show off! ;) We’re running multiple test environments to play with, and one happened to be a LabManager implementation. No modification needed, just flew straight up!

    Reply

  66. Pete’s avatar

    Eric,
    I am using vESXi to run our VMware Academy. The one issue I can’t seem to resolve is getting NTP to work with the vESXi. It did work when I used vESX however the trend from VMware is towards ESXi. I believe this is creating an issue for the vCenter to use AD for authentication. I can get the VM that is running the VC to join the AD but the VC won’t since the hosts are not in sync.

    I’ve also noticed your comments on 64bit OSs, that are “currently” not possible. Are you aware of any potential fix for this?

    Thanks again for all the posts.

    Reply

  67. habibalby’s avatar

    Hello,

    I have vSphere cluster consist of three hosts. Can I setup vESXi4.1 as VM to run on ESX4.0?

    Thanks,
    Hussain

    Reply

    1. Eric Gray’s avatar

      Most likely you will run into problems with the keyboard and won’t be able to install ESXi 4.1 on an ESX 4.0 host.

      Reply

    2. habibalby’s avatar

      Hello,

      Yes, I faced this issue. Any trick for it?

      Another question; i have successfully installed vESX4.1 on pESX4.0 server. And I managed to run nested 32bit VM on the vESX4.1 hosts.

      How can I install the VMWare Tools inside the nested guest OS?

      Thanks

      Reply

  68. ram’s avatar

    Habibalby – I was able to install vESX4.1 inside pESX4.1 however, I am unable to run nested 32bit VM’s inside vESX4.1. How did you do? Appreciate your input pls.

    pESX4.1 is Dell E6400 Laptop Intel VT with 8GB Mem.

    Thanks

    Reply

    1. habibalby’s avatar

      hi Ram,

      you need to enter this line in the Advanced Configuration “monitor_control.restrict_backdoor / TRUE”

      Reply

  69. spice’s avatar

    Hi,

    Can someone advise me if it is possible to install ESX 3.5 as a VM on ESX4? I have a client that has a software product that is only supported esx 3.5 but they platform they have is vsphere 4.0?

    Reply

  70. NiTRo’s avatar

    @spice : yes it’s possible and it works :)

    Reply

  71. spice’s avatar

    Thanks NitRo – How stable is it? would you build for a production environment?

    Reply

  72. habibalby’s avatar

    Hello once again,

    I’m finding this lab is very interested and questions keep raising:)

    What about VMTools on the vESX Server? Do I have to install it? If yes, how to?

    Thanks,

    Reply

  73. Matt’s avatar

    Worked well until I tried to enable either HA or EVC in a cluster. Nor could I start the VMWare Data Recovery appliance on the nested VMs. All of which worked without a problem when done on the bare metal ESXi 4.1 hypervisor.

    Messing with the CPUID would allow the VDR VM boot to start but it wouldn’t get much further into the boot process than a few seconds.

    A little annoying when studying for the VCP not being able to play with those 3 major components. Still not the end of the world.

    Reply

    1. Matt’s avatar

      I tell a lie, increased the memory from 2GB to 3GB and the HA agents were able to start and HA works.

      Reply

    2. Datto’s avatar

      Since you’re just looking at using your nested ESX setup for training, you might look at using the 1.0.x version of VDR under a nested ESXi 4.1 as described similarly by LifeGood in this forum post:

      http://communities.vmware.com/thread/290907

      Datto

      Reply

  74. Pete’s avatar

    For our VMware Academy, I am planning to use a hybrid approach, since vSphere4.1 has hosed several of the Labs: 64bit Data Recovery VM, HA, plus the others I’ve mentioned earlier.

    The hybrid approach is by using SuperMicro’s X8SLI-F motherboard. It is totally compatible with esx/esxi 4.1, and available for <$200. It even includes IPMI.

    I still will use the nested approach for most of the labs, but I thought you might like to know of this mb.

    Reply

  75. Pete’s avatar

    It should be SuperMicro; X8SIL-F

    Reply

  76. Pete’s avatar

    Matt:

    Good on Ya, as they say downunder.

    I bumped the memory up on the nested ESXi hosts then and HA now functions.

    This buys me a some time since I don’t feel as stressed to implement the hybrid setup. To me though the writing is on the wall that unless there is some way to enable VT in the vESX BIOS the requirement for 64bit OS is going to, as Pink Floyd would say, the Wall.

    It is definitely an advantage in having the vESX hosts which is why I still plan to run the training as a hybrid. Being able to build Physical hosts under $1k that can be used as a home or small business Cloud server is tough to beat.

    BTW: There is also an issue with the version of the X8SIL, make sure it is v1.02. This allows for i5 intel processors to work. I’ve have read that it will even work with i7s but I can not confirm this. I’m using neither since the x3400 series will allow upto 32G RAM. You need to verify the RAM is compatible.

    Reply

  77. Jia Ji’s avatar

    Hi Eric,

    I’m so glad I stumbled upon your site! I almost bought a 2nd Server for a home lab setup to test out the HA, DRS, vMotion, vDS, etc. Your article just might have saved me thousands (literally)! I initially tried to test the Enterprise features on a virtual ESXv4 running on a Workstation product hosted on a desktop w/ Intel Core 2 Quad Q6600; however, I quickly realized that even w/ the EVC enabled, I could not do vMotion, etc.

    In any case, here is my current config:
    - 24GB Mem, 1 Quad Core Xeon, 2x2TB HDD, 4 x Gigabit NICs
    - ESXi v4 on internal SD/Flash

    Motivation: Learning/Hobby/Home Virtual Infrastructure (i.e.: Not production)

    Here are my questions:
    - Ideally I’d like to do ESXi on ESXi installation; however, I only have one internal SD
    module w/ 1GB media. Is it possible to do two ESXi installation on ONE 2GB SD media?
    This way, I can boot to Host ESXi from SD as it is currently setup. Then, after I have
    booted, power-on the 2nd instance of ESXi that is also hosted on the same SD card.
    - If the above is not possible or recommended, then I would most likely do ESX on ESX
    installation. Are there any gotchas with this setup with regards to the vDS (virtual
    distributed switch), networking in general, 32-bit/64-bit guest OS support?
    - Also, I’m looking to configure VLAN networking across two (physical & virtual) ESX/i
    hosts. Any suggestions/recommendations/pointers on how I can do VLAN 801q
    tagging for all VLANs through the virtual distributed switch only (i.e.: without a physical
    switch that supports VLAN)? If so, are there any articles on howto setup/configure
    VLANs on ESX?

    Sorry for bombarding you with all the questions, but my recent interest in the ESX product/features has got me too excited.

    Again, I can’t thank you enough for posting some of the most valuable articles on the ‘net (e.g: ESX on ESX, P2V VMotion, etc.).

    Sincerely,

    Jia

    Reply

  78. Dave McLeod’s avatar

    This is a very helpful post. I have one issue though. I was able to install a Windows 2003 server VM inside an ESX VM but the Windows VM can receive IP address via dhcp but cannot ping LAN IP or connect to the internet. I have disabled firewall settings still got the same result.

    Please help.

    Thank you

    Reply

    1. Datto’s avatar

      Dave — you need to engage “Promiscuous Mode – Acceot” on a) the Service Console Management Port Group as well as engage it on b) the Virtual Machine port group.

      Assuming your using lab equipment and not production servers, click on the Properties of your VSwitch in the VIC, select the port group and choose Edit, find the security tab, check the Promiscuous Mode checkbox for Accept/Reject and choose Accept. Then OK out of the dialogs. Then do the same for the other port group/VSwitch (if the VSwitch is different than the VSwitch used for the Service Console).

      That will then allow traffic out from the nested VMs running on nested ESX 4.x hosts.

      Datto

      Reply

  79. Jordan’s avatar

    I was able to do this, but I have an interesting issue when I try to VMotion.

    The physical hosts are Intel Xeon Quad Core CPU’s with Virtualization Technology turned on in the BIOS.

    When I create a VM on the virtualized ESX Hosts, it says I need to enable the Virtualization Technology on my virtualized hosts. Kind of confusing.

    3 Physical 4.1 ESX Hosts (in a cluster)
    3 Virtual 4.1 ESX Hosts (in a cluster)
    1 Virtual Machine on the Virtual 4.1 ESX Host cluster (giving the warning about enabling Virtualization on the host)

    Reply

    1. T-Bone’s avatar

      Jordan,
      In most cases, the vMotion issue can be solved by changing the CPU/MMU Virtualization option on the virtual host. In my test environment running ESXi 4.1 Update 1, I was successful using the 3rd option “Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization”. Your CPU may allow or require a different choice. Try all the available options and test the system’s reaction to each one.

      Reply

  80. Matt’s avatar

    The virtualised Hypervisors no longer have access to the Hardware Virtualisation instructions (i.e. Intel VT and AMD-V) in the bios/cpu since they are only seeing vCPUs presented from the hypervisor on the baremetal. You could try messing with the CPUID Mask in your VM but your best option would probably be to enable Enhanced VMotion Compatibility (EVC) on the cluster.

    http://www.vmware.com/files/pdf/vmotion_info_guide.pdf

    Reply

    1. Jordan’s avatar

      I understand the issue that they no longer have direct physical access to the hardware anymore. I forgot to mention I did try to enable EVC Intel on the cluster but got the error:

      “EVC cannot be enabled because the cluster contains hosts that lack EVC-capable hardware”

      Which also makes sense.

      Reply

      1. Matt’s avatar

        There are some VMs that require access to the hardware virtualised instructions, could that be the issue? I ran into that issue when trying to use the VDR VM, if that is the VM you are trying to run then read back a few posts Datto had a good suggestion. (figured this was worth a mention since its a little unclear from your posts if the error is occurring when you created the VM or during a vmotion within the virtualised cluster)

        Other than that all I’ve got are rhetorical questions, sorry if you’d run though this already: Did EVC work in your Physical hypervisor cluster (i.e. are the physical CPUs happy to work together even before they were virtualised)? Were all the VMs shutdown in the cluster you enabling EVC? Does altering the affected VM’s CPUID Mask change anything?

        Reply

        1. Matt’s avatar

          Decided to do a bit of reading after I checked my lab and found the same issue with EVC. Seems EVC has a requirement of having hardware virtualisation enabled in the bios so its not going to work for a cluster with the nested hosts.
          http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003212

          This of any benefit for the vMotion issue?
          http://vinf.net/2009/06/12/vsphere-esxi-as-a-vm-vmkernel-traffic-not-working/

          Reply

  81. Datto’s avatar

    For those with smaller lab budgets —

    Just wanted to mention that this morning I was able to consolidate eight nested Classic ESX 4.0 hosts each with three lightly used Win2003 VMs as well as two nested XenServer 5.x hosts each with two lightly used CentOS 5.2 VMs on a single physical ESX host. For the physical host I used a 1U Dell SC1435 server with 2x Opteron 2356 CPUs and 16GB of physical memory running ESXi 4.1 from a USB flash drive. Storage for VMs was from one of my Fibre SANs.

    My total cost for the Dell SC1435 server with parts from eBay for it was $512.70 (detailed costs were 2x Opteron quad 2356 CPUs were $84.26 delivered, the Dell SC1435 server was $222.44 delivered and I inventoried the parts not used from this server, 16GB ECC REG memory was $160 delivered, the QLogic PCI-e HBA was $40 delivered and the flash drive was about $6 delivered). I didn’t use the 2x hard drives delivered with the Dell SC1435 server nor the existing DC CPUs that came with the Dell SC1435 server since I’d separately bought two 3rd generation 2356 Opterons for nesting purposes and intended to run the VMs from SAN. I could have just as easily run the nested VMs off the local hard drives and not have needed the Fibre HBA.

    Performance inside the nested VMs running on the nested ESX/nested XenServer hosts appears to me to be the same as when those same VMs ran on nested ESX hosts/nested XenServer hosts that previously ran on three different physical 1 CPU 8GB ESX white box servers. The physical Dell SC1435 server currently is running about 32% for overall CPU, 88% for memory use with about 2GB for vSWP in use (all the nested ESX/nested XenServer VMs get 4GB memory assigned while the nested ESX servers get 1 vCPU and the nested XenServer servers get 2x vCPUs).

    If I’d have used ESXi 4.1 for the nested ESX hosts instead of the Classic ESX 4.0 hosts that would have saved me a bit more memory use but I’d already had the pre-built nested Classic ESX 4.0 VMs built from a different project.

    Reply

    1. Eric Gray’s avatar

      Great setup, thanks for sharing!

      Reply

  82. Paul Grevink’s avatar

    Hi Eric,

    Very good post, especially the part on adjusting the vSwitch and enable promiscuous mode. It saved my day.
    ESXi on ESXi works well without this change. But after installing ESX on ESXi, trouble begun and could only be fixed by changing to promiscuous mode. At this moment the theory behind this change is not completely clear to me…

    Regards,
    Paul

    Reply

  83. Ryan’s avatar

    Has anyone experienced problems with untagged traffic? All my other VLANs / Portgroups work great on both the physical host and nested VMs but I have one untaged portgroup and the VMs are only able to communicate if they are on the physical host. I’m using a distributed switch and have promiscuous mode enabled on all the portgroups.

    Reply

  84. Thom’s avatar

    Has anyone else run into issues when cloning ESXi VMs?

    It seems to work fine initially, but the vCenter managing the virtual ESX hosts gets confused about the datastores. The local datastore on each virtual ESX has the same UUID, leading to problems in vCenter.

    Anyone else seen or solved this? How are the disk UUIDs generated?

    Reply

    1. hussain’s avatar

      because you’ve cloned the esx. I’m facing the same issue on vmware workstation

      Reply

  85. govin’s avatar

    Im trying to install ESX4.0 on the new VM already running in ESX4.0 connect with SAN Storage. The new VM used for the ESX installation space allocated from the SAN Datastore. I need to know during the ESX 4.0 setup on VM ver7.0 would it format only the space allocated to the VM disk or the entire SAN LUN which provisioned to the VM. (Currently I’ve 2 running VM’s in the Datastore) Thanks

    Reply

  86. Eric Gray’s avatar

    vSphere 5 makes this easier by adding a guest OS type for ESX/ESXi. It also supports nested 64-bit guests.

    Check out http://www.vcritical.com/2011/07/vmware-vsphere-can-virtualize-itself/

    Reply

    1. Alain’s avatar

      Is it possible to install vsphere 5 with 2 esxi in a vsphere 4.1.xxx environnement.
      Should i follow the same instruction as in here or is it different for the guest os.

      also is the promiscuous mode really important?

      lafrenal@crim.ca

      Reply

  87. Nitin Reddy Katkam’s avatar

    I use VMs for separating my development environment from my testing environment and use Oracle’s (formerly, Sun’s) VirtualBox. It meets my need though it isn’t stable enough to run on a server. I did run VirtualBox within a Hyper-V virtualized Windows server install and was able to run a VM within a VM, though I used two different products for it.

    Reply

  88. Jeff Messeer’s avatar

    Just finished getting my single new host on ESXi 4.1 U1 up and running. Connected it to my Iomega ix4-200d, and now using iSCSI I have a vCenter VM and two ESXi 4.1 U1 hosts one one layer, and then the second layer I have the virtualized hosts running in a full HA/DRS cluster sharing two iSCSI LUNS. Your original post and the comments here helped me get this thing going – very much appreciated!

    Reply

  89. Richard Clare’s avatar

    I made an ESX vm okay but no matter what I do I can’t connect to it with the VI client and I can’t add it as a host in Virtual Centre. It’s on the same vswitch as the other vms, so basically it’s on the same network as the VirtualCentre box as far as I know, and obviously I can connect to that and that can also connect to the Internet. So I don’t know what’s up. I tried giving it different ip addresses and using a DHCP setting. The virtual machine network is on one physical nic and the management network on the other. Someone suggested moving all the machines over to the management network but I haven’t been able to do it.

    Reply

  90. Matt1’s avatar

    sounds like might have overlooked the promiscuous mode on the vswitch as per the original article.

    Reply

  91. Richard Clare’s avatar

    I added the “monitor_control.restrict_backdoor / TRUE line” like you said but I still get the error “Running VMware ESX in a virtual machine requires the outer virtual machine to be configured for running a VMware ESX guest operating system. You may not power on a virtual machine until the outer virtual machine is reconfigured.” when I try to power on the nested virtual machine.

    Reply

  92. Tom Maloney’s avatar

    Richard,
    Just to verify a few points.
    The “monitor_control.restrict_backdoor” parameter is added not to the nested VM, but to the Advanced configuration of the VM that contains your virtual host.
    The Parameter name is “monitor_control.restrict_backdoor” and the Value is “TRUE”. Check to make sure there are no leading or trailing spaces – this can happen a lot if you copy/paste from the article. Also, make sure to NOT use the “/” character.
    If this all checks out and you still have this problem, download and open the .VMX file of the virtual host in a text editor like Wordpad. The line should look like this –

    monitor_control.restrict_backdoor = “TRUE”

    Reply

  93. Jack’s avatar

    I have imported a virtual machine(win xp 32 bit) to the virtual ESX host, but the network connection of the win xp is not available, I try many ways, could any one please tell me how to create valid network connection with other machines?

    Thanks in advance

    Reply

    1. Tom Maloney’s avatar

      Jack,

      Not sure exactly the symptoms from your description, but I’ll try.

      First, you need to treat the virtual host as though it is a normal ESX host. This means you need to create any port groups on this server with the proper VLAN IDs and Port Group name. The port group name must be the same as what is defined on the other systems.
      Next, the uplinks – the vmnics – on your virtual host need to have all of your VLANs trunked to it. The easiest way to accomplish this is to use promiscuous mode. You need to set this on the physical host.
      Example – host ESX01 is the physical host, ESX02 is running inside a virtual machine on ESX01. You would create a vSwitch (vSwitch1 for example) on ESX01 and put it in promiscuous mode. The VM that runs ESX02 would have one or more virtual NICs tied to ESX01 – vSwitch1. At ESX02, you would make sure that your vSwitch is joined to vmnic0 and create a VM port group on the proper VLAN for your XP VM.
      Go back and review that section of the posting. You may have used promiscuous mode on the wrong switch.

      Reply

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>