The VMware ESXi 4 64MB Hypervisor Challenge

In a previous article, I answered the question: If VMware ESXi 4 is so small, why is it so big? It’s quite clear now that the disk footprint of VMware ESXi 4 is less than 60MB.  But to really drive the point home, I wanted to demonstrate that VMware ESXi 4 could boot and run from a tiny 64MB flash device, so I asked Olivier Cremel, the inventor of ESXi, if that was feasible.  He said it was — and gave me advice on how to set it up.  This article shows you how.

Requirements

  • The VMware ESXi 4 dd image file (Vladan’s site may help)
  • A 64MB flash drive — check your junk drawer
  • A Linux machine — make sure you have mkfs.vfat
  • This fdisk script — save it as a txt file

Extract necessary components from the large image

Use dd to copy the first 1.5MB of the boot partition and the files that make up ESXi.

# dd if=VMware-VMvisor-big-175625-x86_64.dd of=esxi-boot-part skip=32 count=3000
3000+0 records in
3000+0 records out
1536000 bytes (1.5 MB) copied, 0.036249 seconds, 42.4 MB/s
# mkdir tmp
# mkdir esxi
# mount -o loop,offset=$((8224*512)) VMware-VMvisor-big-175625-x86_64.dd tmp/
# cp tmp/* esxi
# sync
# umount tmp/

Prepare the flash device

Plug in your flash device and use dmesg to determine the device id (/dev/sdk in this example).  Run fdisk on it and delete all partitions – very important.  Then, run my fdisk script to create the required partition structure.  If this doesn’t work on your flash drive due to different size/geometry, you will need to create the partitions yourself — refer to the table at the end of these instructions for reference.

# fdisk /dev/sdk < fdisk.txt

Write data to the flash device

Use dd to write the saved boot partition to flash.

# dd if=esxi-boot-part of=/dev/sdk4
3000+0 records in
3000+0 records out
1536000 bytes (1.5 MB) copied, 1.31462 seconds, 1.2 MB/s
# sync

Then create a FAT filesystem on the flash drive, mount it, and copy the ESXi files.

# mkfs.vfat /dev/sdk5
mkfs.vfat 2.11 (12 Mar 2005)
# mount /dev/sdk5 tmp/
# cp esxi/* tmp/
# sync
# umount tmp/

Double-check your partition layout

If your disk partition does not look like this, something went wrong.

# fdisk -l /dev/sdk
 
Disk /dev/sdk: 65 MB, 65273856 bytes
3 heads, 42 sectors/track, 1011 cylinders
Units = cylinders of 126 * 512 = 64512 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdk1              26        1011       62118    5  Extended
/dev/sdk4   *           1          25        1554    4  FAT16 <32M
/dev/sdk5              26        1011       62097    6  FAT16

Test it out

If you followed all of the instructions above, you should be able to boot from that flash drive.

There you have it — a fully functional VMware ESXi 4 system entirely contained on a 64MB flash device. Of course, this procedure is entirely unsupported, so please do not use a system like this in production. Please!

Tags: , , , ,

21 comments

  1. Christian Mohn’s avatar

    Oh!! This is great info, will have to try this at work!

  2. NiTRo’s avatar

    You means ESXi is french ?!

  3. Mike Laverick’s avatar

    This great. It really does definitively burys any M$ FUD surround this argument. But I wonder if we and Microsoft have been looking at ESXi for the completely wrong angle. On the plane over to San Francisco this week – myself and Tom Howarth began to discuss ESXi short-comings. Anyway, I’m think when I get back to the UK next week – that I will sit and dissect the problems with ESXi. Interested in bouncing some ideas about?

    1. Eric Gray’s avatar

      I’d be interested, of course. Probably see you at VMworld.

    2. asus77x’s avatar

      i try it before and failed. now my flash disk only shown to have 4MB. is there anyway to wipe the flash disk? originaly it’s 2GB.
      thanks.

    3. NiTRo’s avatar

      you can use HP Drive Key Boot Utility to wipe it

    4. Mark Richards’s avatar

      I realize this is a late comment but:

      Can you elaborate on why you performed some of these steps? For example- why do you skip the first 32 blocks? Why 1.5MB? is that the size of the files in the partition? The first partition itself is actually 4 MB in size so why 1.5MB?

      1. Eric Gray’s avatar

        The partition table entries are not in disk order. Run fdisk -u -l VMware-VMvisor-big-175625-x86_64.dd to see what I mean.

        On my “64MB” USB stick I did not have access to a full 64MB — typical for storage devices of all kinds, so I had to trim that partition down a bit. 1.5MB was chosen arbitrarily and it worked.

        1. Mark Richards’s avatar

          I’ve been able to get esx to boot on a key using this method- but no changes I make to the system are being saved to the key. If I reboot- the changes are lost. Are you able to save changes to the key across reboots?

          1. Eric Gray’s avatar

            This is an exercise to show that the core functionality of ESXi is fully contained in 64MB and not 1GB as some have claimed.

            1. Mark Richards’s avatar

              I understand it’s meant as a demo but a demo that doesn’t really work isn’t nearly as impressive as one that doe. In my case I have lots of 256 MB USB keys I’d like to give to people to get them to try esxi- but it’s a lot less useful if the config doesn’t get saved.

              I’m not really sure why it wouldn’t save the config- everything is definitely there- esxi boots- vm’s run- etc. It just seems like the memory filesystem isn’t being written out to disk.

              I actually had a 256MB key so I went a little further with my test: I set up a partition for VMWare tools and installed those to the key (I left out Vsphere client as I can download and install the latest version from vmware’s site easily enough and at 111MB it would have taken up too much room). The tools worked fine and let me install to the vm’s I tested.

              I’d still like to figure out the save issue if anyone has any ideas.

            2. Kevin Brooksby’s avatar

              Nice work Eric. Great post. Used this method (a la a bit modified to extract the VMware-VMvisor.dd.bz2 on a Linux box) and I also used a 128 MB flash stick I had laying around. I must admit this is very cool to prove the core thin hypervisor is indeed a very small footprint and I appreciate your explanations in your two-part articles explaining the use of the other partitions. Doesn’t seem to load as quick (but this may be because it is a 1.x USB device and not 2.x complaint like most of my 1-2 GB ESXi usb sticks I test with) but all functionality is there minus of course tools/vSphere client (web interface give a cute blank page as I assume it is missing tomcat portlet)/etc. Booted up a xppro VM that I recently installed the openssh for windows package on for test I was doing for a friend (I know it seems so bastardized) and yanked the 128MB ESXi u1 stick and to my immediate surprise my ssh session into that xppro box stayed running as did the VM. Must have been because the ESXi hypervisor is loaded into memory. Powered off my laptop running ESXi that joins my two way cluster and approx 45 seconds later HA fired the VM back up on esx2 node. Very cool indeed! Also noted the settings do not stick but that is no surprise with this stripped/slim ESXi install.

              Keep up the good work!

            3. VAroon’s avatar

              Can this work on a normal laptop for testing purposes.But with no CPU AMD-V

            4. patrick’s avatar

              can it install on a usb flash, then use dd to backup the sda4 and sda5,

              and then use this 2 patition backup file to restore to a 64MB flash with your fdisk script?

              1. Eric Gray’s avatar

                Patrick, you could use a 1GB USB as an intermediate source as you suggest. That’s initially how I approached this, but figured it would be easier to not require two different USB sticks.

                Eric

                1. patrick’s avatar

                  the sd*4 seems large the 1.5M and the data inside it seems less than 512K?
                  dd if=VMware-VMvisor-big-175625-x86_64.dd of=esxi-boot-part skip=32 count=3000 should not contain all the sd*4? It is work?
                  if the data is less than 500K, can I use dd if=VMware-VMvisor-big-175625-x86_64.dd of=esxi-boot-part skip=32 count=1000 ?
                  has you test the VMware-VMvisor-Installer-4.0.0.Update01-208167.x86_64.iso, is it can store in a 64M disk?

                2. patrick’s avatar

                  can it use the same way to copy the file to sd*4?
                  mount -o loop,offset=$((32*512)) VMware-VMvisor-big-175625-x86_64.dd tmp/

                  1. Eric Gray’s avatar

                    Patrick,

                    Please keep in mind that this is just a proof-of-concept and not intended for real-world use. The point was to demonstrate that ESXi is very small despite the “huge” 1GB flash requirement. Given the difficulty in even finding a sub-1GB USB flash device these days, there is not much value in getting it to run on 64MB.

                    When I put together the prototype, I stopped when I hit on a repeatable formula that works. I’m sure there are ways to optimize, feel free to tinker around and post your findings if you come up with a cool solution.

                    Thanks for reading VCritical.

                    Eric

                  2. juan’s avatar

                    Hello

                    Here is another way to do it:
                    I use the fantastic boot loader grub4dos and boot ESXi with this entry:

                    title Vmware ESXi4
                    find –set-root /grldr
                    kernel /ESX4/vmkboot.gz
                    module /ESX4/vmk.gz
                    module /ESX4/sys.vgz
                    module /ESX4/cim.vgz
                    module /ESX4/oem.tgz
                    module /ESX4/license.tgz
                    module /ESX4/mod.tgz

                    It boot from a pendrive with NTFS filesystem.
                    You only need to install grub4dos on the pendrive
                    and take these files from the ESXi dd image.
                    This way you can boot ESXi, Windows, Linux, etc. from ONE pendrive.
                    Greetings!

Comments are now closed.