Newsletter 
RSS Feed 
Follow
Running VMware ESX inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. It is pretty common to do this on VMware Workstation nowadays — the first public documentation of this process that I know of was published by Xtravirt a couple of years ago.
But what if you prefer to run ESX on ESX instead of Workstation?
You may be pleased to know that the GA build of ESX 4 allows installing ESX 4 as a virtual machine as well as powering on nested virtual machines — VMs running on the virtual ESX host. You can even VMotion a running virtual machine from the physical ESX to a virtual ESX — on the same physical server!
VMware vSphere 4.1 UPDATE: VMware ESXi 4.1 has a keyboard issue when virtualized on an ESX 4.0 host. In order to virtualize ESXi 4.1, the underlying host must be 4.1. However, ESX 4.1 classic will work on ESX 4.0.
The extra tweaks to make it all work are minimal, and I will show you how without even opening up a text editor.
After installing ESX 4 onto your real hardware, configure as desired and enable promiscuous mode on a vSwitch:
Create a new VM with the following guidance (choose “Custom”):
- Virtual Machine Version 7
- Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
- 2 VCPUs, 2GB RAM
- 2 NICs – e1000
- LSI Logic Parallel
- New disk – reasonable size greater than 10GB for ESX classic
After you have the VM ready, simply attach a VMware ESX 4 ISO image, power on, and install ESX as a guest OS.
After installation, add the new virtual ESX to vCenter 4 and create a new VM.
If you do not need to run VMs on your virtual ESX, you can stop there. However, if you try to power on that nested VM, you will see the following error:
To prevent this, just one tweak is needed:
- Shut down the virtual ESX VM
- Click Edit Settings
- Click the Options tab
- Click Advanced / General / Configuration Parameters…
- Click Add Row
- For the Name/Value enter: monitor_control.restrict_backdoor / TRUE
The above procedure is just an alternative to hand-editing the .vmx file — if you prefer to do it that way, feel free.
Now you are ready to power your virtual ESX VM back on, as well as the nested VMs. This capability should come in handy as you start investigating the new features of vSphere 4.
UPDATE: In order to take snapshots of a VM running ESX, a quick configuration change is necessary on the physical host .
You may be interested in this related post where a VM is migrated between the physical and virtual ESX hosts.
Related posts:
Tags: ESX, installation, Technical, vSphere
-
Eric: Good stuff. I think this is a great way for administrators to get started with the base functionality of the product without allocating a tremendous amount of hardware. This won’t help too much with upgrades, but will definitely help with the base product!
Cheers.
-
Thx you very much . it was very use full for me .. i don’t find any Youtube video on this ..
-
Thanks, this works well. I do have a problem loading the tools on the vm’s I create on the virtual esx. Does anyone have any ideas.
Thanks
-
i’ve also noticed that you can’t clone a vm from one virtual esx to another virtual esx.
-
You should be able to come up with a way to do this, but there are some issues — such as duplicate MAC addresses — that need to be dealt with.
Feel free to chime in if anyone out there has some tips for Mike.
-
-
-
-
Eric,
Great post. I’ve been struggling to achieve this since long time with no luck. Unfortunately though its not working on the ESX 4.0 RC, so I guess I’ll have to wait for the GA. Besides, I believe you need to add this inside the nested VMs config: mx.allowNested = “TRUE” ? or it’s not required any more?
thanks
-
Eric, great post!
-
Rick, you are probably right about not helping with upgrades from ESX 3.x, but going forward this environment could be used to test the ESX 4 patches and upgrades. Actually, I’m told that ESX 3 will also run in a VM like this but it is hardware dependent, and I have not tried it myself.
Hany, that additional configuration element is no longer needed — it was in the past for certain configurations. By the way, internally VMware has been running ESX on ESX for a long time. A special build was required, though.
Anton, spasebo vam! It’s very cool to see my articles translated into Russian. Especially if you take a look at my background on the About page…
-
Hi Eric,
I am using esx on esx. I have a number of w2k3 vm’s running under it. I’m having problems loading the tools. It says Vmware tools should be installed inside a virtual machine.
Any ideas? Is there another switch I need to set.
Thanks
Mike
-
-
Eric,
Would you mind testing out installing ESX 3.5 as a VM on ESX4? This would be huge for us here! -
Great post. I would have posted something similiar myself but I was told I would be in breach of the NDA agreement I’d signed…
-
Justin, you can run ESX 3.5 in a VM on certain hardware — AMD Barcelona and Shanghai, probably Intel Nehalem.
-
Pingback from ESX as a VM - nothing new @ A. Mikkelsen on May 20, 2009 at 12:11 pm
-
Eric,
I’m a bit confused here. I know that the ESX VM is 4.0, but what about the underlying physical ESX host, does it have to be ESX 4.0 also, or it can be ESX 3.5?
I just tired the ESX 4.0 GA as a VM on ESX 3.5U3 but it didn’t work for me, the VM starts then a I get a blank screen and nothing happens, few seconds later the VM shutdown by itself and no errors are given.
Any ideas?
Thanks
-
In my example, the physical host is also running ESX 4.
-
-
Hi Michael,
In my testinstallation, vESX4.0(GA,RC1) is running on physical ESX3.5U4(but I have trouble with vswif0 after installation), and on W2K3+VMWorkstation 6.5.2. -
Eric,
I try this with a ESXi4 final, but I can install and boot only Esxi3.5(very unstable).
I can not install ESXi4, ESX3.5 and ESX4(Final)(PSODs and more PSODs).
Where I download this version (GA)?
My hardware is a whitebox(Athlon64 X2 7750 extreme)
Sorry for my english, thanks. -
thanks Yue Yao for the note, in my case the vESX4.0 boots/run fine, and i have no issues whatsoever with the networking, it’s just the VM running within the vESX4.0 that is not work for me, I’m using physical ESX3.5U3, so I’ll upgrade to U4 and see how it goes…
I just hope Eric won’t kick us out of here :p
-
Astor: by GA I am referring to the official (final) release — not beta/RC.
Unfortunately, it’s hard to say why folks are experiencing mixed results without digging in deeper.
-
To get ESX 4 hosts working as VMs within your ESX 3.5 environment do the following:
1) Create the VMs to run ESX4 on an ESX 3.5 host.
2) Add these VMs to the inventory on an ESX4 host.
3) Build the virtual ESX 4 hosts as above.
4) Once built, these VMs should run in the ESX3.5 environment.Sadly, vmware tools do not work. Any ideas?
-
Pingback from Twitted by dwaler on May 29, 2009 at 3:30 am
-
Hi,
Can you start a nested windows x64 VM under the vESX 4.0 (VM) ?
-
Rene, I do not believe 64-bit nested VMs are possible at the present.
-
-
Can you verify this for me? Would like to know if window 2008 x64 would run (without hyper-v)
-
rene,
I can verify x64 VM’s aren’t running within the vESX 4.0 (VM). Only 32 bit hosts boot. More info can be found here. Hope this answers your question.
-
I came a bit lat to this post because I was away of my Island at holidays in France. It sounds great all this.
In the pas Xtravirt published a white paper for ESX 3.0 and 3.5. I could learn a lot from this. Now They did the same thing again… For ESX 4.0.
-
Heiko, thanks for pointing that out for Rene.
Welcome back, Vladan.
-
Great info, I was able to get 4.0 working in 3.5U4, once I modified the vSwitch.
If anyone is interested, I got it working with 768 MB of memory (instead of 2GB)- documented here http://infinitedisorder.com/?p=158 -
Akabdog – cool, thanks for the contribution.
-
http://www.jhangra.com/blog/2009/07/how-to-install-esxvsphere-4-0-in-vmware-workstation-6-5-2/
But has a video which helped me quite a bit.
-
Hi,
I installed ESXi 4.0 base host & couple of ESXi servers in there as a VM.
Now I am unable to run nested windows 32 bit VM’s on those ESXi servers.
All I get is a blank console while booting the nested VMs.
I even tried deploying a VM from Marketplace, however, the VM gets deployed, gets poweredON, but only a blank console to look at.
No error messages as such while powering ON the nested VM.Thanks in advance.
Ali -
Hi,
Great article. I’ve been able to setup a complete vSphere test rig on a single €200 ML115.
One issue I have is that I can’t add hosts to a dvSwitch. I have available NICs but the Add Hosts list is blank. Has anyone had any luck with this?
Thanks,
Neil.
-
Hi,
I just answered my own question. dvSwitches are only available with the Enterprise Plus and evaluation licenses, so NFR licenses don’t work as they are just Enterprise.
Neil
-
eric: I was successfull installing virtual ESX 4.0 inside physical ESX 4.0. But I am not able to power on VM’s inside Virtual ESX. I had did the configuration changes as you have mentioned above. Could you please help me in this?
-
Rahul, I suspect it is your CPU. What is the exact vendor and model info?
-
Eric: I tried it in Dell power edge server and it worked….Thanks a lot…
keep posting this kinda stuff….its really helpfull.-
Good to know, thanks.
-
-
Hi Eric,
Great Web site and articles. I was going to buy more hardware to build a test environment. Now I am going to try to create another ESX 4.0 server in my physical ESX 4.0. I hope I will be able to test vMotion with this setup.Thanks for the great work.
Regards,
Shan -
Ali – I am having the same problems with the blank/black console from vSphere (running on the Host. The issue persists in both ESX and ESXi 4.0. I suspect that it may be a Windows 7 issue (host). Like you, I have tried an appliance (2 actually, a Turnkey LAMP app and VA-ntop).
Next I plan to try:
Import a Win XP VM and try to RDP to it from within Win7.
Build a pWinXP, install vSphere and try that.
Lastly (is that a word?) Install Wkstn on the pXP machine, install vESX and try to get a console from pWin7.I will report back, but it might be a week or 2 as I have several pESX boxes to build this week.
-
Hello
Has anyone been able to successfully view the console of a nested VM regardless of OS. My VMs also have blank/black consoles…
cheers
gogogo5 -
Yes …I had installed WIN2K3 R2 as a VM GOS inside my virtual ESX 4.0 . I didnt find any issue as you are facing. I was able to use my nested WIN2K3 VM console.
-
Hi Eric,
please could you talk a bit about your hardware setup and the performance? Are you able to run a cluster of two virtualized ESX to make some VMotion, HA etc.?
-
Sven,
You can absolutely exercise most of the features of vSphere with a 2-node virtual ESX cluster. VMotion, HA, Host Profiles, vDS — everything should work the same way. Performance of the nested VMs is not stellar, but you can at least get a feel for all of the functionality. Check out the various posts linked in these comments to see what others are doing with virtual ESX systems.
Eric
-
-
Hi Eric,
I have 2 x ESX4 hosts running on the same ESX3.5 AMD Dell box. I have configured iSCSI and vmotion on the 2 ESX vm’s. I can successfully see the shared storage on the 2 boxes, and created a vm that is runnign w2k3. Problem i have is that they will not see each other via the service console or vmption network. Do you know of any further network tweaks? I have already set the vswitch promiscuous mode to accept.
Thanks
-
Deon, it’s too hard to say for sure. Interesting that you are using ESX 3.5 as the host, did you try with ESX 4?
-
-
Hi,
Have try this install on a HP DL360 G4, ESX 4 is running fine.
But when I try to start the virtual ESX4 I receive the following error:
“This version of ESX requires a CPU capable of 64-bit operation.”
CPU’s are EM64T as the ESX4 hardware install is working.
-
Hi Goris,
I know I am too late in answerin this Question. But I think rest users would be able to get the benefit out of the same.
Once u create the VM…go to the Option …….check in the option where in you get the BIOS screen at Power on.
2- Now go the advanced tab in the processor you have 1.4 & 1.1. select 1.1 version….
press F10..you are ready to go.
It worked for me(MY White box is Q9550 + S3210 SHLC+ 8 GB).-
Hi sripati,
I tried this with my DELL PE 2850 and the host esx is working with ESX 4, and when i followed ur steps above i get the same error still “This version of ESX requires a CPU capable of 64-bit operation.”.
Then i down graded the esc to 3.5 U5 and still i get the error.
But I can run ESX4 on the DELL tin still what do i need to do please let me know anyone so I can start working on my lab and get my VCP soon.
many thanks
DA
-
-
-
have run “CPU Identification Utility” from vmware.
It seems to the CPU’s can not run 64bit as virtual
Supports 54-bit Longmode: Yes
Supports 64-bit VMware :No -
Hi,
I cannot run VMs on “virtual” ESX4, blank console and 100% CPU,
is-it related to AMD opteron ? I just bought a poweredge sc1435 for a lab … Nicolas
-
I have seen this issue with not enabling Virtualization Technology on the AMD processor in the BIOS – double check and let me know if that works!
-
-
I’m running 7 ESXi 4 to test HA stuff…
Thanks again for this good post Eric -
Eric,
I am running VM in ESX 4.0 Evaluation version on my desktop, but unable to power on the VM in it. I just get a black/blank screen and never get any screen even though it says that the VM is powered on. I have an ISO for windows 2003 for which I dont get any installation wizard.
Guru.
-
Hi Eric
Do you have an Intel or Amd ?, you MUST have virtualisation instructions enabled ( VT / in the bios).
Nicolas
-
-
Pingback from My Home Lab Setup « Rick Vanover's Blog on January 20, 2010 at 10:51 pm
-
Nicolas,
I have an AMD 64 bit, enabled the bios setting (NX for AMD) for virtualization.
Guru
-
Ok, on amd there is a special generation level you must have !!, even with VT enabled it does’t work.
You can search if you have a second generation or a third one this link, you MUST have at least a third one !!
http://www.cpu-world.com/CPUs/K10/AMD-Third%20Generation%20Opteron%208356%20-%20OS8356WAL4BGD.html
-
Thanks for your contribution, Nicolas.
-
-
I have an Athlon 64.. Not sure about the generation.
Guru
-
Eric, This is great info.
I had discovered this for 3.5 last year, Check posts from vmware Academy Partners,
One added issue we have been having with our dell 2950’s is that the install would hang on formatting the vmfs.We have been able to resolve this by using a RDM for the disk of the esx VM.
Pete
-
Eric,
I have a HP DL385 G2 with AMD Opteron Processor. I have enabled the Virtualization in BIOS. This machine has enough RAM and DASD.
I was able to install ESX 4 on the bare metal machine
I am trying to get more familar with vSphere and to get familar with other features I needed another ESX server so I followed your post and was able to successfully install a virtual ESX 4i instance within the ESX 4 server. I have successfuly added both the ESX servers in the test cluster in vCenter.
I have also added a bare metal pentium 4 OpenFiler machine which is working as iSCSI storage. I have successfully configured and openfiler storage is visible to the ESX4 and virtual ESX4i instance
virtual ESX4i instance allows me to create virtual machines. However when I power them on I am not able to see anything in the console. When I power on the same machine and migrate it to the physical ESX host and I can see the console.
I am a virtualization newbie who is trying to learn … any assistance will be greatly appreciated.
Thanks!
-
Hi Eric… Thx for this post!
I executed the 2 tweaks and was able to install an ESX3.5 server and an ESX 4 serverThe ESX4 server fails after a reboot. it fails on vsd-mount ….
Any suggestions?
The ESX3.5 runs fine
Thanks
Fred -
It looks like we need another tweak for snapshot usage….. because that was the issue.
The ESX 3.5 reboots fine (with snapshot in use) but the ESX4 fails on vsd-mountAny suggestions?
Tx
fred -
I’m seen the same issue when I create snapshots on Virtual ESX 4s. I think it has to do something with the fact that the COS is on the same vmfs disk as the one you use for regular vmstorage. I’ve Google and found the issue, but I’m not sure if anybody has found a workaround yet.
-
FYI, I had problems getting ESX 4.0u1 to install under ESX 4.0 on an HP DL680c G5. When the install would start formatting the disks, vmkfstools would fail with status 65280, print the vmkfstools command options, then say “Error: not supported”. It turns out I only had the problem when the guest disk was on a volume on a NetApp filer via NFS. Putting the volume on direct-attach disk led to a successful installation.
-
The snapshot issue may be addressed with an undocumented configuration setting. Please see http://www.vcritical.com/2010/02/taking-snapshots-of-vmware-esx-4-running-in-a-vm/
-
Thanks for this advanced configuration setting Eric!
My Physical ESX host is ESX4i and sadly it is an unknown Option in there…C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-advcfg.pl –server nl01sol
vh10 –username root -g /COW/PermitVmfsOnRedoHierarchy
Enter password:
Option /COW/PermitVmfsOnRedoHierarchy not found.C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-advcfg.pl –server nl01sol
vh10 –username root -s 1 /COW/PermitVmfsOnRedoHierarchy
Enter password:
Option /COW/PermitVmfsOnRedoHierarchy not found.-
Since this is an unsupported/undocumented configuration setting, it cannot be controlled via remote API. The good news is that you should be able to just log into the “tech support mode” console of ESXi and run the same command that works on the service console.
In other words, go to the ESXi console, press ALT+F1, type in the word “unsupported” (blindly, as it is not echoed to the screen), enter your password when prompted, and then run the esxcfg-advcfg command.
-
-
Thanks again Eric! i went to “unsupported “service console” on the physical ESX4i and execu ted esxcfg-advcfg -s 1 /COW/PermitVmfsOnRedoHierarchy
made a snapshot and could startup without any problem. -
Hi Eric,
I have installed Vmware Vsphere on vmware workstation and I am able to create the HA & DRS cluster with FREENAS Shared storage but my problem is when I create the VM and try to power on the VM with the ISO or any other medium, I am getting blank black screen and I am not getting any kind of error but just the blank screen.
I even changed the points like monitor_control.restrict_backdoor = TRUE etc.
When I poer on the ESX host VM I am still getting the error that I can’t run the NESTED VMs.
-
Prashant,
Could it be that you have an older AMD CPU? If you notice, some of the comments above have had similar issues and one reader felt it was AMD-specific. I have used AMD Shanghai CPUs for this and everything worked properly.
Eric
-
-
Eric,
Once again — good stuff. I set up ESXi V4 on a Dell R900 W/128GB Ram and 8 CPUs. We have 15 or so test VMs running on it.
With that said I downloaded the 60 trial of ESX 4.0.u1 and vSphere. Installed a 3 host cluster and turned on HA and DRS. Added 6 guest VMs on the virtualized ESX and all works great, systems are a bit slow but that’s what we get for multilayer hypervisors.
My question is — has anyone got Fault Tolerance to work? When I turn it on I get a hardware compatibility error. It seems that FT wants VT turned on in the bios, but the VMware bios of the guest ESX server goes not have this option. The backend storage is IB so there is no delay there. Any ideas?
I need to give a presentation demo to the higher ups and they are interested in FT. -
Hi Eric,
Thanks for this post!
I have installed ESX 4 on a white-box (Dell 9200) and it works like a charm.
Now I’ve followed yours instructions to install ESXi4 as a VM on it.
And I got a PSOD “No place on disk to dump data”.
Any suggestions ?Thanks
-
Hi,
I have two esx servers as guest os on my physical esx host but the two esx guest are unable to communicate each other.
Please help me in acheiving this feature.Thanks
Bhaskar. -
Hi Eric we have the same name! Anyway, if you need a good company to buy any VMware vSphere 4 products, a good place is my company ITO Solutions. Here is a link to our site for VMware ESX http://www.itosolutions.net/Server_Datacenter_Virtualization_s/523.htm
-
Hi Eric, do you have any clue on how to run 64bit VM inside the ESX that is running over a virtual machine?
-
Currently not possible…
-
-
Hi,
I have installed ESX 4.0 on workstation 7. the ESX has 22 gig VMFS datastore. I am trying to create a new VM in ESX and dont know how to boot from ISO to install the OS. After creating the VM when I go to connect cd/dvd it did not detect the OS in dvd drive. Do i have to copy the ISO to datastore first if yes then how i can do that or I can only create VM using the VC 4.any help appreciated
-
Ok, so I have a problem with my esxi 4 “update 2″ guests running under the same esxi 4 version for the host. The problem is that the guest esxi servers do not have correct time despite having ntp setup just like the host.
The host esxi server has been pointed to pool.ntp.org and shows the correct time via the vshpere client. The guest/s esxi servers are pointed to the same exact ntp server “pool.ntp.org” but after restarting the service just like I did on the host the time does not change and is off. I manually set the time on the guests to the correct time but after 8-10 hours the time is faster on the guests by about 2 minutes and some change. Are others experiencing time difference on there nested esx/i vm’s? I figured it would use the ntp server and adjust it time correctly just like the host did. I also verified they could resolve and communicated with the ntp server …so that can be taken out of the equation.Thanks,
Mike
-
Great article, help me established a test ENV for Lab manager, thanks!
-
Hello Eric
My Host OS is windows 07, on which I have installed VMware server and ESX 4.0 inside it
Following is the order
Win07Ultimate (Host OS)–> VMware-server-2.0.2-203138 –>ESX4.0 –> win 2008 VM
I tried above solution of setting up the Vswitch in promiscuous mode and adding the line “monitor_control.restrict_backdoor / TRUE”
But still I get the error message “You may not power on a virtual machine in a virtual machine”
Is there any solution if the ESX is not directly installed on the hardware as the host OS
Regards,
Anantha -
Hello all,
I tried to install the ESX v4 server on ESX v4 server as described in the beginning of this article. The installation cannot even take place.
Error code:
CosSup: 406: vmkernel not loaded: cannot PSOD – Basically not supported microcodes.My cluster configuration is with EVC AMD Opteron Generation 3. The exact processor type is Ams Opteron Quad-Core 8389.
Installation of ESX v3.5 is working, but after reboot I have the same error all the time even with different VM settings.
Error code:
…….initialization of vmkernel failed, status 0xbad001e – Microcodes problem again.Can someone help?
Thanks.
-
Does any one tried to move ESX 4.0 Host VMs to Another ESX4.0 Host.
I hve Dell 2950 2.6 GHz 8x Core, want to shift my ESX 4.0 Linux VMs from this one to DELL 2950 2.0 GHZ processor.Any idea?
I have VCenter VM also on the same ESX Host1.
Thanks for advise in advance.MAZ
-
MAZ – You should have no problems moving VM’s. My questions would be is how exactly are you thinking of doing this? Are you wanting to use vMotion? Are these host in a HA/DRS cluster? Answering these questions would help me better understand what answer your looking for.
Thanks!
-
-
Trackback from Kasraeian on August 30, 2010 at 3:21 pm
-
Just wanted to note that anyone “experiencing” issues with blank screens and such on AMD platform CPUs be sure to check Vritulization Enabled on the Processor settings in the BIOS. I know this may sound kind of insulting but believe it or not I missed this on my system and yes I felt like a total newbie. I could easily run VM’s in workstations but had random issues with nested VM’s. Just double check and make sure that you weren’t pulling a “chad” and NOT saving the BIOS configuration…. would’ve saved me a headache..
Disclaimer
Sponsors
101 comments
Comments feed for this article
Trackback link: http://www.vcritical.com/2009/05/vmware-esx-4-can-even-virtualize-itself/trackback/