Newsletter 
RSS Feed 
Follow
Running VMware ESX inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. It is pretty common to do this on VMware Workstation nowadays — the first public documentation of this process that I know of was published by Xtravirt a couple of years ago.
But what if you prefer to run ESX on ESX instead of Workstation?
You may be pleased to know that the GA build of ESX 4 allows installing ESX 4 as a virtual machine as well as powering on nested virtual machines — VMs running on the virtual ESX host. You can even VMotion a running virtual machine from the physical ESX to a virtual ESX — on the same physical server!
The extra tweaks to make it all work are minimal, and I will show you how without even opening up a text editor.
After installing ESX 4 onto your real hardware, configure as desired and enable promiscuous mode on a vSwitch:
Create a new VM with the following guidance (choose “Custom”):
- Virtual Machine Version 7
- Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
- 2 VCPUs, 2GB RAM
- 2 NICs – e1000
- LSI Logic Parallel
- New disk – reasonable size
After you have the VM ready, simply attach a VMware ESX 4 ISO image, power on, and install ESX as a guest OS.
After installation, add the new virtual ESX to vCenter 4 and create a new VM.
If you do not need to run VMs on your virtual ESX, you can stop there. However, if you try to power on that nested VM, you will see the following error:
To prevent this, just one tweak is needed:
- Shut down the virtual ESX VM
- Click Edit Settings
- Click the Options tab
- Click Advanced / General / Configuration Parameters…
- Click Add Row
- For the Name/Value enter: monitor_control.restrict_backdoor / TRUE
The above procedure is just an alternative to hand-editing the .vmx file — if you prefer to do it that way, feel free.
Now you are ready to power your virtual ESX VM back on, as well as the nested VMs. This capability should come in handy as you start investigating the new features of vSphere 4.
You may be interested in this related post where a VM is migrated between the physical and virtual ESX hosts.
Related posts:
Tags: ESX, installation, Technical, vSphere
-
Eric: Good stuff. I think this is a great way for administrators to get started with the base functionality of the product without allocating a tremendous amount of hardware. This won’t help too much with upgrades, but will definitely help with the base product!
Cheers.
-
Eric,
Great post. I’ve been struggling to achieve this since long time with no luck. Unfortunately though its not working on the ESX 4.0 RC, so I guess I’ll have to wait for the GA. Besides, I believe you need to add this inside the nested VMs config: mx.allowNested = “TRUE” ? or it’s not required any more?
thanks
-
Eric, great post!
-
Rick, you are probably right about not helping with upgrades from ESX 3.x, but going forward this environment could be used to test the ESX 4 patches and upgrades. Actually, I’m told that ESX 3 will also run in a VM like this but it is hardware dependent, and I have not tried it myself.
Hany, that additional configuration element is no longer needed — it was in the past for certain configurations. By the way, internally VMware has been running ESX on ESX for a long time. A special build was required, though.
Anton, spasebo vam! It’s very cool to see my articles translated into Russian. Especially if you take a look at my background on the About page…
-
Eric,
Would you mind testing out installing ESX 3.5 as a VM on ESX4? This would be huge for us here! -
Great post. I would have posted something similiar myself but I was told I would be in breach of the NDA agreement I’d signed…
-
Justin, you can run ESX 3.5 in a VM on certain hardware — AMD Barcelona and Shanghai, probably Intel Nehalem.
-
Pingback from ESX as a VM - nothing new @ A. Mikkelsen on May 20, 2009 at 12:11 pm
-
Eric,
I’m a bit confused here. I know that the ESX VM is 4.0, but what about the underlying physical ESX host, does it have to be ESX 4.0 also, or it can be ESX 3.5?
I just tired the ESX 4.0 GA as a VM on ESX 3.5U3 but it didn’t work for me, the VM starts then a I get a blank screen and nothing happens, few seconds later the VM shutdown by itself and no errors are given.
Any ideas?
Thanks
-
In my example, the physical host is also running ESX 4.
-
-
Hi Michael,
In my testinstallation, vESX4.0(GA,RC1) is running on physical ESX3.5U4(but I have trouble with vswif0 after installation), and on W2K3+VMWorkstation 6.5.2. -
Eric,
I try this with a ESXi4 final, but I can install and boot only Esxi3.5(very unstable).
I can not install ESXi4, ESX3.5 and ESX4(Final)(PSODs and more PSODs).
Where I download this version (GA)?
My hardware is a whitebox(Athlon64 X2 7750 extreme)
Sorry for my english, thanks. -
thanks Yue Yao for the note, in my case the vESX4.0 boots/run fine, and i have no issues whatsoever with the networking, it’s just the VM running within the vESX4.0 that is not work for me, I’m using physical ESX3.5U3, so I’ll upgrade to U4 and see how it goes…
I just hope Eric won’t kick us out of here :p
-
Astor: by GA I am referring to the official (final) release — not beta/RC.
Unfortunately, it’s hard to say why folks are experiencing mixed results without digging in deeper.
-
To get ESX 4 hosts working as VMs within your ESX 3.5 environment do the following:
1) Create the VMs to run ESX4 on an ESX 3.5 host.
2) Add these VMs to the inventory on an ESX4 host.
3) Build the virtual ESX 4 hosts as above.
4) Once built, these VMs should run in the ESX3.5 environment.Sadly, vmware tools do not work. Any ideas?
-
Pingback from Twitted by dwaler on May 29, 2009 at 3:30 am
-
Hi,
Can you start a nested windows x64 VM under the vESX 4.0 (VM) ?
-
Rene, I do not believe 64-bit nested VMs are possible at the present.
-
-
Can you verify this for me? Would like to know if window 2008 x64 would run (without hyper-v)
-
rene,
I can verify x64 VM’s aren’t running within the vESX 4.0 (VM). Only 32 bit hosts boot. More info can be found here. Hope this answers your question.
-
I came a bit lat to this post because I was away of my Island at holidays in France. It sounds great all this.
In the pas Xtravirt published a white paper for ESX 3.0 and 3.5. I could learn a lot from this. Now They did the same thing again… For ESX 4.0.
-
Heiko, thanks for pointing that out for Rene.
Welcome back, Vladan.
-
Great info, I was able to get 4.0 working in 3.5U4, once I modified the vSwitch.
If anyone is interested, I got it working with 768 MB of memory (instead of 2GB)- documented here http://infinitedisorder.com/?p=158 -
Akabdog – cool, thanks for the contribution.
-
http://www.jhangra.com/blog/2009/07/how-to-install-esxvsphere-4-0-in-vmware-workstation-6-5-2/
But has a video which helped me quite a bit.
-
Hi,
I installed ESXi 4.0 base host & couple of ESXi servers in there as a VM.
Now I am unable to run nested windows 32 bit VM’s on those ESXi servers.
All I get is a blank console while booting the nested VMs.
I even tried deploying a VM from Marketplace, however, the VM gets deployed, gets poweredON, but only a blank console to look at.
No error messages as such while powering ON the nested VM.Thanks in advance.
Ali -
Hi,
Great article. I’ve been able to setup a complete vSphere test rig on a single €200 ML115.
One issue I have is that I can’t add hosts to a dvSwitch. I have available NICs but the Add Hosts list is blank. Has anyone had any luck with this?
Thanks,
Neil.
-
Hi,
I just answered my own question. dvSwitches are only available with the Enterprise Plus and evaluation licenses, so NFR licenses don’t work as they are just Enterprise.
Neil
-
eric: I was successfull installing virtual ESX 4.0 inside physical ESX 4.0. But I am not able to power on VM’s inside Virtual ESX. I had did the configuration changes as you have mentioned above. Could you please help me in this?
-
Rahul, I suspect it is your CPU. What is the exact vendor and model info?
-
Eric: I tried it in Dell power edge server and it worked….Thanks a lot…
keep posting this kinda stuff….its really helpfull.-
Good to know, thanks.
-
-
Hi Eric,
Great Web site and articles. I was going to buy more hardware to build a test environment. Now I am going to try to create another ESX 4.0 server in my physical ESX 4.0. I hope I will be able to test vMotion with this setup.Thanks for the great work.
Regards,
Shan -
Ali – I am having the same problems with the blank/black console from vSphere (running on the Host. The issue persists in both ESX and ESXi 4.0. I suspect that it may be a Windows 7 issue (host). Like you, I have tried an appliance (2 actually, a Turnkey LAMP app and VA-ntop).
Next I plan to try:
Import a Win XP VM and try to RDP to it from within Win7.
Build a pWinXP, install vSphere and try that.
Lastly (is that a word?) Install Wkstn on the pXP machine, install vESX and try to get a console from pWin7.I will report back, but it might be a week or 2 as I have several pESX boxes to build this week.
-
Hello
Has anyone been able to successfully view the console of a nested VM regardless of OS. My VMs also have blank/black consoles…
cheers
gogogo5 -
Yes …I had installed WIN2K3 R2 as a VM GOS inside my virtual ESX 4.0 . I didnt find any issue as you are facing. I was able to use my nested WIN2K3 VM console.
-
Hi Eric,
please could you talk a bit about your hardware setup and the performance? Are you able to run a cluster of two virtualized ESX to make some VMotion, HA etc.?
-
Sven,
You can absolutely exercise most of the features of vSphere with a 2-node virtual ESX cluster. VMotion, HA, Host Profiles, vDS — everything should work the same way. Performance of the nested VMs is not stellar, but you can at least get a feel for all of the functionality. Check out the various posts linked in these comments to see what others are doing with virtual ESX systems.
Eric
-
-
Hi Eric,
I have 2 x ESX4 hosts running on the same ESX3.5 AMD Dell box. I have configured iSCSI and vmotion on the 2 ESX vm’s. I can successfully see the shared storage on the 2 boxes, and created a vm that is runnign w2k3. Problem i have is that they will not see each other via the service console or vmption network. Do you know of any further network tweaks? I have already set the vswitch promiscuous mode to accept.
Thanks
-
Deon, it’s too hard to say for sure. Interesting that you are using ESX 3.5 as the host, did you try with ESX 4?
-
-
Hi,
Have try this install on a HP DL360 G4, ESX 4 is running fine.
But when I try to start the virtual ESX4 I receive the following error:
“This version of ESX requires a CPU capable of 64-bit operation.”
CPU’s are EM64T as the ESX4 hardware install is working.
-
have run “CPU Identification Utility” from vmware.
It seems to the CPU’s can not run 64bit as virtual
Supports 54-bit Longmode: Yes
Supports 64-bit VMware :No -
Hi,
I cannot run VMs on “virtual” ESX4, blank console and 100% CPU,
is-it related to AMD opteron ? I just bought a poweredge sc1435 for a lab … Nicolas
-
I’m running 7 ESXi 4 to test HA stuff…
Thanks again for this good post Eric -
Eric,
I am running VM in ESX 4.0 Evaluation version on my desktop, but unable to power on the VM in it. I just get a black/blank screen and never get any screen even though it says that the VM is powered on. I have an ISO for windows 2003 for which I dont get any installation wizard.
Guru.
-
Hi Eric
Do you have an Intel or Amd ?, you MUST have virtualisation instructions enabled ( VT / in the bios).
Nicolas
-
-
Pingback from My Home Lab Setup « Rick Vanover's Blog on January 20, 2010 at 10:51 pm
-
Nicolas,
I have an AMD 64 bit, enabled the bios setting (NX for AMD) for virtualization.
Guru
-
Ok, on amd there is a special generation level you must have !!, even with VT enabled it does’t work.
You can search if you have a second generation or a third one this link, you MUST have at least a third one !!
http://www.cpu-world.com/CPUs/K10/AMD-Third%20Generation%20Opteron%208356%20-%20OS8356WAL4BGD.html
-
Thanks for your contribution, Nicolas.
-
-
I have an Athlon 64.. Not sure about the generation.
Guru
-
Eric, This is great info.
I had discovered this for 3.5 last year, Check posts from vmware Academy Partners,
One added issue we have been having with our dell 2950’s is that the install would hang on formatting the vmfs.We have been able to resolve this by using a RDM for the disk of the esx VM.
Pete
-
Eric,
I have a HP DL385 G2 with AMD Opteron Processor. I have enabled the Virtualization in BIOS. This machine has enough RAM and DASD.
I was able to install ESX 4 on the bare metal machine
I am trying to get more familar with vSphere and to get familar with other features I needed another ESX server so I followed your post and was able to successfully install a virtual ESX 4i instance within the ESX 4 server. I have successfuly added both the ESX servers in the test cluster in vCenter.
I have also added a bare metal pentium 4 OpenFiler machine which is working as iSCSI storage. I have successfully configured and openfiler storage is visible to the ESX4 and virtual ESX4i instance
virtual ESX4i instance allows me to create virtual machines. However when I power them on I am not able to see anything in the console. When I power on the same machine and migrate it to the physical ESX host and I can see the console.
I am a virtualization newbie who is trying to learn … any assistance will be greatly appreciated.
Thanks!
Disclaimer
64 comments
Comments feed for this article
Trackback link: http://www.vcritical.com/2009/05/vmware-esx-4-can-even-virtualize-itself/trackback/