Emulex OneConnect / HP Madness

If you have Emulex OneConnect FCoE adapters in your HP blades, you're in for a real treat when you try and upgrade to vSphere 5.0 and find a completely unsupported and unworkable mess. Let's get down to the details:

First off, the inbox be2net driver delivered from VMware is no good, make sure that you download the right driver from VMware's website ( 4.0.355.1 ). The inbox driver causes some serious problems if you use more than a single VLAN on the network end. You can install the driver using VUM, or you can use VMware Image Builder and inject the new driver into the installation media.

Now for the fun part, in order to use the 4.0.355.1 driver, you need a compatible firmware loaded on the Emulex OneConnect adapters, and 2 months after the release of vSphere 5.0, HP is still unable to provide a 'supported' firmware. The closest thing that you can find from HP is the 4.0.360.7 firmware:

Unfortunately the release notes don't paint a rosy picture for users of FCoE:

IMPORTANT: Emulex Firmware Version 4.0.360.7 enables support for HP ProLiant G7 BL servers and Virtual Connect with VMware vSphere 5.0. Support for FCOE functionality on the CNA is not included in this firmware. This firmware release does not support PXE Boot on a ProLiant server with AMD processors. This firmware release also does not support hp CloudSystem Matrix/Matrix Operating Environment. NCDE 10.45, which includes updated Emulex firmware, is targeted to be released in the mid-November timeframe to address the remaining critical customer issues such as: FCOE Failover not recovering, EDD looping or Hang, PXE not responding, slow PXE performance, PXE failures with Hypervisor, Red Screen, and Intermittent Ping loss.

On top of that, if you'd like to be able to use CIM providers from both HP, and Emulex you're out of luck as there is a problem with the HP CIM provider ( and the way they overlay libraries ) that conflicts with the Emulex CIM providers. Which leads to a pretty uncomfortable truth. It's pretty clear that VMware doesn't do any validity checking of 3rd party VIBs, it's also clear that vendors don't check their modules against any other vendors modules ( even if they're partnered ), which leaves a pretty big support gap as there is no one that you can call to get this kind of mess straightened out. So even though all of these components are on the 'compatability matrix' provided by VMware, there is still no supported way to get them all working harmoniously. I can say that VMware has been helpful at driving the conversations between the vendors, but the lack of even rudimentary testing by VMware is troubling.

HP maintains that they should have a OneConnect firmware available in a few weeks that should have support for FCoE ( 3 months after GA release of vSphere 5.0 ), but they're also maintaining that they won't construct a new VIB that doesn't do the overlay wrong until Feburary of 2012.

As a side note, it looks like HP has produced a publicly consumable depot for VMware VIBs. If you point VUM at http://vibsdepot.hp.com you should be able to have it automatically check for driver / CIM provider updates without manually checking their website, a trend that I would like to see other 3rd parties follow.

ESXi v5 features.

Here is a quick summary of the new features for ESXi v5.

http://vmguy.com/wordpress/index.php/archives/1674

He also has some information about vCloud Director 1.5.

http://vmguy.com/wordpress/index.php/archives/1734

Thanks to Dave Lawrence (vmguy) for the detailed list.

ESX Software ISCSI and Jumbo Frames.

How to enable the Software ISCSI Initiator (iSCSI Software Adapter) and use Jumbo Frames on the network.

1. Ensure that your network is configured for Jumbo Frames. If you have Cisco devices you must use the "MTU 9XXX" setting on every interface between your storage and your ESX Server. This incudes switchport and trunk type interfaces.
Example :
interface TenGigabitEthernet1/1
mtu 9198

2. Ensure that your SAN can support Jumbo Frames. If you are using a white box server running OpenFiler or like software Raid device, just ensure that the software is configured for jumbo frames and that the physical network card within the hosting server supports jumbo frames. Just google the adapter information you read from the physical card in the server. Here is a quick list of Intel adapters that do not or partly support jumbo frames.

http://www.intel.com/support/network/sb/cs-001911.htm

3. Install and configure the vSwitch and VMkernel that will have access to your storage device. I have a separate network for my ISCSI storage and another for my Service Console + VMKernel for vmotion, HA, and DRS. So, for my configuration, I install a second VMKernel with a different IP address for my storage network.
- Within the Vcenter GUI, click on your host, click on the configuration tab, and click Networking.
- Click the Add Networking link and a wizard will launch.
- Select the bullet for Virtual Machine and click next.
- Select the physical network adapter that your Jumbo Frame enabled storage network is connected to and click next.
- Enter a name for this network. Use something simple to type such as "iscsinetwork" and click next and finish.
Its very important to note what vSwitch number you just created. It will show on your GUI vSwitch# next to the name you used above.

*** note. You could create the VMKernel with this GUI but it will not enable Jumbo frames.

Now the fun part:












- SSH into your ESX Host and log in using a root type account. If you want to log in as root you need to make sure you edit your /etc/ssh/sshd_config file and restart the service before attempting to ssh into the server.

- Once you are logged in, execute these commands:

esxcfg-vswitch -m 9000 vSwitch3
- The above command will configure the vswitch you created to use an MTU size of 9000. (jumbo) Be sure to change the vswitch number to match your new vswitch.

esxcfg-vmknic -a -i 10.10.10.4 -n 255.255.255.248 -m 9000 iscsinetwork
- the above command will create a VMKernel with ip, subnet, and MTU of 9000 on the vswitch you created with name iscsinetwork. Be sure to change the name to match what you used above.

*** note. You can test your complete Jumbo Frame network path by using this command at the command prompt.

vmkping -s 8900 10.10.10.2

Be sure to change the ip to match the IP of your storage device that is hosting your LUNs. I use 8900 MTU size because of the packet header of Ethernet. If you get successful 8900 mtu pings then everything is working great!

iSCSI Configuration time:
4. Within the Vcenter GUI, click on your host, click on the configuration tab, and click Storage Adapters.







- You should see the iSCSI Software Adapter. Select it and click properties to the right.
- Check the status of the Initiator. If it says Disabled, click the configure button, check the Enabled box and click OK.
- Click the CHAP button and configure if you will use CHAP or not. I recommend using CHAP with a large and complex username and password but if this if for test you can leave it blank. It must match what your storage device is configured for.
- Click the Dynamic Discovery tab and click Add.

*** note. If you have time, I would configure one LUN at a time. Complete step four and five completely for each single LUN to make things easy.

- Enter the DNS or IP address of your Storage Device. If you are using a global CHAP configuration for all of your LUNs, then you do not need to configure CHAP again here. It will pull the information from the parent setting you configred above. If you have a different setting for the LUN you are configuring now, you can enter the different CHAP information here.
- Click OK and click Close. It will ask you if you would like to rescan for new storage. Click Ok and it will start the process to look for new storage.

: This process will only connect the host to your storage device. You still need to create the VMFS on the new storage for it to be usable and visible.

5. Within the Vcenter GUI, click the Storage link on the same host you configured your new iSCSI LUN.
- click the Add Storage link to the right and it will launch a wizard.
- select disk/LUN and click next.
- You should see your new empty storage container. If the window is empty, you will need to start troubleshooting why the storage did not connect to your Storage Device.

*** note. If you configured a single LUN, it is easy to identify which LUN you attached and you can easily name it. If you connected more than one LUN then you will have a harder time attempting to identify which Storage LUN is connected to which VMFS. Possible, just harder.

- select your Storage and click next.
- review the information and click next.
- Enter a name of the storage container as it will appear within Vcenter.
- Select your block size and click finish.
1M = 256GB max single VMDK size
2M = 512GB max single VMDK size
4M = 1024GB max single VMDK size
8M = 2048GB max single VMDK size

*** note. You will need to delete and reformat the VMFS container if you wish to change this setting later.
- When the task completes you should see the new storage in your Datastores window.

Vmware ESX/ESXi 4 and Windows Disk Alignment

In the last few months, I've heard way too many storage venders pitch the words "disk alignment" to VMWare admins. So, I would like to share some quick information on how to "align your disk" so you can feel special or possibly improve your disk IO.

A hard thing for home brew or a small business is to buy fast and reliable storage. A Fiber channel setup at home would make the spouse scream and attempting to sell it to the boss is difficult when "they sell hard drives at BestBuy for fifty bucks!" Sure, ill plug that USB 2.0 laptop hard drive into our Prod ESX server and use it as shared storage. Shared means we can unplug it and share it with other people right?.... (enough wasted ranting)

This is a two block process and can only be done before data is placed within the storage area. Meaning, your storage LUN must be formatted with VMFS version 3 or later and your Windows OS drive needs to be partitioned correctly before the OS is installed. Attempting this after data exists on the VMFS or once the OS is installed is difficult to impossible.

A) The VMFS must be aligned with your storage device blocks. This process assumes you are running ESX/ESXi version 4 or newer. See the link at the bottom of this write-up if you are using version 3 or older.
Your LUN is aligned when you let the Vcenter GUI create and format the volume.

1. Within your Vcenter GUI, select your host and click on the configuration tab.
- I'll assume you've attached the storage to your Host using the "Storage Adapter" link. The way you connect the Storage LUN to the host is different between types of storage so I'll let you complete that step on your own.

2. Click the Storage link and click Add Storage



3. Select if your Storage is a LUN or NFS and click next.

4. If you are configuring a SAN LUN, select the available LUN, review the layout, and name your storage. Some NFS configurations will require path information to configure.

5. Select the Block size for your storage.
1M = up to 256GB (a single VMDK of this size, not total LUN size.)
2M = up to 512GB
4M = up to 1024GB
8M = up to 2048GB ( a single 2TB drive within a vm! Exciting)

6. When you are at the end of the wizard, you can review your disk layout before clicking Finish. Here you can see that it will be formatted with VMFS version 3.



7. In theory, your VMFS storage location is now aligned correctly on your SAN. Review your Storage venders best practice documents to ensure no special settings are further required to tune your VMFS storage.

Now lets Create the VM and align the OS drive!
B) The OS VMDK is aligned only within the VM.

1. Create or deploy your VM with correct OS VMFS size. You can create additional aligned volumes after the OS is installed.

2. Boot your VM with your OS/PE CD.

The following steps are for Windows 2003:
- You cannot complete the correct align command that exists on the Windows 2003 OS ISO. A Windows 2008 installer or PE works great.
- You will need to download a PE boot disk or use a Windows 2008 OS disk.
- I spent some time to try and provide a Windows PE boot disk or download link but with so many versions and possible security issues, I'll let you figure this part out. Using a Windows 2008 bootable ISO is the best way to go if you have one.
- If using a Windows 2008 OS disk, boot from the ISO and click "next" at the first prompt. On the second screen click "Repair your computer." This will let you install storage drivers or just click next. This should bring you to a window where you can click "Command Prompt." If you are using a WinPE disk, attempt to get to a command prompt.

3. Once at the command prompt type "diskpart" and press enter.
4. Type "list disk" to show a list of all of the disks attached to the VM.
5. Type "select disk 0" assuming the OS disk is 0 (Zero)
6. Type "create partition primary align=64"
7. At this point you are done. If the disk already has a partition do the following:
7.1. Select the disk with "select disk 0" the type "list partition"
7.2. Type "select partition 1" (enter)
7.3. Type "delete partition" (enter)
7.4. Type "list partition" (enter) to ensure no partition exists.
7.5. Now type "create partition primary align=64"(enter)
8. Exit and reboot your VM.
9. Load the correct OS disk and install the OS. When prompted to select or create a partition, use the existing partition and format it to your liking.



The following steps are for Windows 2008:

No special disk alignment is required, the OS will align the disks you add without any user interaction. The OS installation and disk management GUI will both align the disk correctly before and after the OS is installed.

The following steps will help align a Windows 2003 disk inside of windows.
1. Within your OS, open a command prompt.
2. Once at the command prompt type "diskpart" and press enter.
3. type "list disk" to show a list of all of the disks attached to the VM.
4. type "select disk 1" assuming the new disk is 1, it may be 2!
5. type "create partition primary align=64"
6. Now you need to format the new volume.
7. Right click on My Computer and click Manage.
8. Click on "Disk Management"
9. Your new volume should automatically popup. If not click Actions menu and click Rescan Disks.
10. Right click on the new volume and select format.
11. Select "32K" in the "Allocation Unit Size" window.
12. Don’t forget to check "perform a quick format" if you want to keep your thin provision configuration.
13. You may also need to assign it a drive letter so right click the volume and click "Change drive letter and paths" Select to add a drive letter and click OK.
13. The volume is now aligned and ready to use!


If you want to align a Linux disk, review the vmware document below and adapt it to your flavor of Linux!


Quick reference locations:
How to align your VMFS and OS in ESX version 3.
http://www.vmware.com/pdf/esx3_partition_align.pdf

Here is an Equallogic Document if interested:
http://communities.vmware.com/servlet/JiveServlet/download/948245-9390/VMwareFAQv1-27.pdf

Any Virtual Workstation on the Go.

Project: To have a Windows OS travel with you to any PC.

The Windows OS is very picky with drivers and hardware changes making it impractical to install on a USB stick and expecting any functionality when you plug it into another computer.

Requirements: Two USB sticks greater than 2GB (4GB or larger is best) , a working computer, and internet access.

You could complete this project with a single USB stick but because USB has slow disk I/O, using two helps greatly. I used a Windows 7 PC to download the software and run the utilities.

1) Download Ubuntu.
- Go to http://www.ubuntu.com/desktop/get-ubuntu/download
- Click the download button for the iso. I used "ubuntu-10.10-desktop-i386.iso"









2) Port the OS to a USB stick.
- Download this utility: (the Ubuntu download site links to this URL as well. It will ask for an admin user to run)
http://www.pendrivelinux.com/downloads/Universal-USB-Installer/Universal-USB-Installer.exe - It helps if you save the above download to the same location you downloaded the Ubuntu ISO.
- Plug in the first USB stick, this will hold your Boot OS. (Note the drive letter assigned to it)
- If you are using Windows, quick format the USB stick with the default settings to help with the process and reduce popups.
- Run the Universal-USB-Installer.exe
- Within the Utility, click "I agree" if you agree to the License.
- Select "Ubuntu 10.10" from the top drop down.
- Browse to the location where you downloaded your ISO. (it may auto detect the ISO and not let you browse, this is good!)
- Select the Drive letter of your first USB stick.
- Move the "Persistent file size for storing changes" up to at least "1000" This is needed to install Vmware Player. If you skip this setting your changes with-in Ubuntu will not be saved when you shutdown.
- If you are using a large USB stick, you can move this setting as high as you want and install other Linux utilities. Do not move it to 100% because the software might have issues allocating the full amount of free space after the OS install. 80% works best for me.
- Click create. (You may see a few messages about it erasing everything on the USB stick. )




















3) Download Vmware Player for the Ubuntu Install.
- Download Vmware player. http://www.vmware.com/products/player/
- Once you click the download button you will need to login or create a new account. It should be free.
- Be sure to download the Linux Binary bundle. I used the second link "Vmware Player for Linux 32bit"
- File name I used was "VMware-Player-3.1.3-324285.i386"
- Once downloaded, copy this file to your second USB stick.
- If you want the documentation saved, download it here: http://www.vmware.com/pdf/vmware_player310.pdf

4) Boot your PC into Ubuntu:
- Place your first USB stick into your PC and boot.
- You may need to edit the BIOS to enable USB booting or change the device boot order. Some machines let you select the boot device by pressing F9 or F12 on boot. If you have problems booting and get an error close to "I/O error on device fd0" you will need to disable the floppy in the computer BIOS.

5) Configure the OS to reduce System Config issues:
- Once booted into the OS, run a terminal window. ( Click Applications at the top left of the screen, Select Accessories, and then Terminal.)
- Type "sudo passwd root" (enter)
- Enter your new password and press enter.
- Remove the floppy driver. Within the same Terminal window type "sudo nano /etc/modprobe.d/blacklist" (enter)
- inside the editor type "blacklist floppy" Press (control+c) then y and enter to save and exit.
- We must blacklist the floppy driver because the boot will hang on some PCs that do not have a physical floppy present.

6) Install Vmware Player.
- Place your second USB stick in the Ubuntu booted PC.
- Run a terminal session. (Click Applications at the top left of the screen, Select Accessories, and then Terminal.)
- Navigate to the second USB stick location where Vmware Player is located. Type "cd /media" (enter)
- Type "ls" (enter) to list the name of your second USB stick.
- Type "cd *"(enter) (where * is the name of your second USB stick found above.)
- Type su (enter) and enter your root password.
- Type "sh VMware-Player-3.1.3-324285.i386" (enter) (you can type "sh VM" then press TAB to finish the name)
- When the launcher GUI loads you will answer two questions about Product updates and Sending stats to Vmware. Select Yes or No and click Next then install.
- Click close when the installer is finished and you can exit out of the terminal window.

7) Create Your Windows VM:
- You can do this two ways. (Create a new VM or convert an existing)
1) You can run VMWare player within your running Ubuntu system and create a new VM there.
- Run Vmware player (Click Applications at the top left of the screen, System Tools, Vmware Player)
- Select "Create a New Virtual Machine"
- Select where the OS for the VM will be. You can select to install the OS later or copy an ISO of your OS to a USB stick and select that location in the GUI now.
- Select your OS and click next
- Name your VM and select the storage location. The storage location can be your first USB stick if you used a large one and have enough free space or you can plug in your second USB stick and select it as the storage location. I would recommend using a second USB stick to avoid any IO issues with your Ubuntu boot and VM.
- Select the storage size of your VM. Keep in mind the free space on your USB storage device. Click Next and finish.
- You will now see your VM in the list of the left of the player window, if it did not auto start from the GUI, click it and click Play Virtual Machine.
- From here you can install the OS and any custom software you want to run.

2) You can use Vmware converter and import an existing Windows machine or ESXi VM.
- Download Vmware converter. http://www.vmware.com/products/converter/
- Once you click the download button you will need to login or create a new account. It should be free and is 120MB.
- Install the Vmware converter software (next, next, accept, next, local installation, next, install….. finish)
- Run the application.
- Click "convert machine"









- in the drop down you can select the type of workstation you want to convert. I used an existing ESXi VM so I selected "Vmware Infrastructure Virtual Machine"
- enter the ESXi server name, root username and the password and click next. (you may get a popup about a cert error. You can ignore)
- Select your VM from the list and click next.
- In the drop down select "VMware Workstation"
- Name your VM or leave it as the default.
- Select the location you want to store the VM. We want the VM to exist on your second thumb drive. You can insert your second USB stick and browse and save it there or save it to your hard drive and then move the folder to your thumb drive later. I created a folder c:\temp\Xpmachine and browsed to this location and click next.
- Configure your VM before its converted.
- Click "Data to copy








- In the window to the right, click the drop down and select "2GB Split not pre-allocated" This is required because some errors may happen within Vmware player on Ubuntu if the VM hard disk file is greater than 2GB.
- Click Devices and set the number of CPU and amount of RAM. I left it 1CPU and 512MB Ram because I never know how much ram the system I may plug my USB sticks into will have.
- click Next, and Finish.
- When the process is complete, move the folder to your second USB stick. It should contain a few files. A *.vmdk and a *.vmx. The number of vmdk files will depend on the size of the VM you converted split into 2GB files.
- Plug the second USB stick into your booted Ubuntu desktop and run Vmware player (Click Applications at the top left of the screen, System Tools, Vmware Player)
- Click "Open a Virtual Machine"
- Browse to your second USB stick, open the folder you saved your converted VM in and double click on the *.vmx file.
- Click "Play Virtual Machine" to launch your VM!

My Ultimate Goal:
I have a small USB hub with four ports on it. I have converted a set of VMs onto different Micro SD cards and placed each VM set into a micro SD reader sitting in each port of the USB hub. It’s a workstation pack on a stick. Each VM you want to load, just plug in the memory stick/VM to the USB hub and run it.























Here is a 4GB Micro SD with micro USB reader for $7.50
http://www.newegg.com/Product/Product.aspx?Item=N82E16820161297&cm_re=microSD-_-20-161-297-_-Product
Here is a 32GB USB stick for $38.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16820220467&cm_re=usb_memory_32gb-_-20-220-467-_-Product


Future Goal:
As technology moves toward USB 3.0 and larger micro SD cards, we will have more IO bandwidth and be able to run more VMs per memory stick.

Notes: You may want to create a 32bit and 64bit bootable Ubuntu USB stick. I've ran into some AMD PCs that are very picky what flavor of Ubuntu you attempt to run. I recommend to have a 4GB USB stick with 32 bit Ubuntu with Vmware player installed and another 4GB USB stick with 64 bit Ubuntu and VMware player installed. Then, if your VM is on a separate memory stick you can just match the pair when you boot on the PC.

Adding drivers to the ESXi installer, and installed images

As I noted in a previous post, you can use the vibddi tool to inject drivers into a customized image of ESXi, what I didn't mention was that the ESXi installer would still not have access to those drivers until the installer environment (sys.vgz) is also modified to have the drivers available at installation time. I found that out this previous friday when I attempted to install ESXi 4.1 Update 1 on a number of HP BL460c G7 servers that use the be2net NIC driver which isn't included in the base installation image of ESXi. Here are the steps to injecting the driver into both the installable image (imagedd.bz2), and the installer image (sys.vgz). For this process you're going to need access to the auto-deploy appliance, the ESXi 4.1 Update 1 install iso, the be2net driver iso, and you'll need to locate the binutils RPM ( described later ).

If you're looking to solve the same problem ( be2net driver on HP hardware ) there are 2 options that are easier than what I'm about to describe below:

1. Wait for HP to create a HP OEM version of the ESXi 4.1u1 .iso, download and use that
2. Download the HP OEM ESXi 4.1 iso, install it on your system and then upgrade it to 4.1u1 using one of the supported update methods.

I believe it goes without saying that this is unsupported by VMware, or me. For the intrepid, keep reading.

Log into the auto-deploy appliance as vi-admin and get a root shell
#>sudo -s

Copy the contents of the ESXi 4.1 Update 1 iso image to the auto-deploy appliance, place it in the /esxi41u1 directory.
#>mkdir /esxi41u1
#>mount -o loop /home/vi-admin/VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso /mnt
#> cp -pr /mnt/* /esxi41u1
#> umount /mnt

We also need the driver .iso image that we want to inject into the ESXi 4.1u1 installable image. We need to mount it and then copy the offline bundle from it:
#> mkdir /driver
#> mount -o loop /home/vi-admin/vmware-esx-drivers-net-be2net_400.2.103.377.0-1vmw.2.17.249663.358277.iso /mnt
#> cp /mnt/offline-bundle/SVE-be2net-2.103.377.0-offline_bundle-358277.zip /driver
#> umount /mnt

Uncompress the imagedd.bz2 image
#> bunzip2 imagedd.bz2

Inject the necessary drivers into the installable image, note that a new be2net driver was released 2/11/2010 ( same day as 4.1 update 1). Check the VMware download page to see if there are any newer drivers.
#> vibddi -i imagedd -n -o SVE-be2net-2.103.377.0-offline_bundle-358277.zip
Setting up configuration for esxupdate
This may take a few seconds...
Checking bootbank filesystems (pre-install)
Installing offline bundle: /home/vi-admin/SVE-be2net-2.103.377.0-offline_bundle-358277.zip
Update operation completed successfully
Checking bootbank filesystems (post-install)

(optional, install more offline VIBs, I install PowerPath PP/VE, HP NMI Sourcing Driver, and the HP CIM providers, not shown )

Verify that the driver is now installed into the image ( in this case vmware-esx-drivers-net-be2net )
#> vibddi -i imagedd -q

oem-vmware-esx-drivers-net-vxge (400.2.0.28.21239-1OEM)

oem-vmware-esx-drivers-scsi-3w-9xxx (400.2.26.08.036vm40-1OEM)

vmware-esx-drivers-net-be2net (400.2.103.377.0-1vmw.2.17.249663)

vmware-esx-firmware (4.1.0-1.4.348481)

CMPI-1.0

VMW_CMPI_CPP-1.0.5

vmkapi_1_1_0_0

vmknexus1kvapi-1-4

vmkepsecapi_1.0.0.0

DriverAPI-9.0

DriverAPI-9.1

vmware-esx-tools-light (4.1.0-1.4.348481)

#> bzip2 imagedd
#> md5sum imagedd.bz2
b9b9d2d724a3c04b6e1341c8f8507367 imagedd.bz2

Edit /esxi41u1/imagedd.md5 to match the new md5sum using the text editor of your choice
#> nano /esxu41u1/imagedd.md5
b9b9d2d724a3c04b6e1341c8f8507367 imagedd.bz2

Next, the be2net driver needs to be injected into the installer environment ( sys.vgz ). To accomplish this, you need to have the 'ar' utility installed on your auto-deploy appliance and it doesn't come pre-installed. Download the newest x86_64 binutils RPM for Centos 5 ( which is what the auto deploy appliance is based on ). I used the following RPM: binutils-2.17.50.0.6-14.el5.x86_64.rpm.

#> rpm -ivh binutils-2.17.50.0.6-14.el5.x86_64.rpm
warning: binutils-2.17.50.0.6-14.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID e8562897
Preparing... ########################################### [100%]
1:binutils ########################################### [100%]

Unzip the driver bundle:
#> unzip /driver/SVE-be2net-2.103.377.0-offline_bundle-358277.zip
Archive: SVE-be2net-2.103.377.0-offline_bundle-358277.zip
inflating: metadata.zip
inflating: vmware-esx-drivers-net-be2net-400.2.103.377.0-1vmw.2.17.249663.x86_64.vib

Extract the vib archive using the 'ar' utility that we installed.
#> cd /driver
#> ar vx vmware-esx-drivers-net-be2net-400.2.103.377.0-1vmw.2.17.249663.x86_64.vib
x - debian-binary
x - control.tar.gz
x - data.tar.gz
x - short.rpm
x - descriptor.xml
x - sig.pkcs7

Unpack the data.tar.gz
#> tar -zvxf data.tar.gz
./
./etc/
./etc/vmware/
./etc/vmware/pciid/
./etc/vmware/pciid/be2net.xml
./etc/vmware/init/
./etc/vmware/init/manifests/
./etc/vmware/init/manifests/vmware-be2net.mf
./usr/
./usr/lib/
./usr/lib/vmware/
./usr/lib/vmware/vmkmod/
./usr/lib/vmware/vmkmod/be2net.o

The three files that we're interested in are be2net.o ( the driver ), be2net.xml ( the pciid ), and the vmware-be2net.mf ( manifest ). Now we just need to unpack the installer environment (sys.vgz), add these files and then repack the installer environment.


Unpack the installer environment
#> mkdir /systemp
#> cp /esxi41u1/sys.vgz /systemp
#> cd /systemp
#> gunzip --suffix .vgz sys.vgz
#> vmtar -x sys -o sys.tar
# >tar -xvf sys.tar
... ( lots of output of all the extracted files ) ...

Copy the 3 required files to add the be2net driver
#> mkdir -p /systemp/etc/vmware/init/manifests/
#> cp /driver/etc/vmware/pciid/be2net.xml /systemp/etc/vmware/pciid
#> cp /driver/etc/vmware/init/manifests/vmware-be2net.mf /systemp/etc/vmware/init/manifests
#> cp/driver/usr/lib/vmware/vmkmod/be2net.o /systemp/usr/lib/vmware/vmkmod

Repack the installer environment
#> rm sys
#> rm sys.tar
#> tar cvf sys.tar *
... ( lots of output of all the files that are being tar'd up ) ...
#> vmtar -c sys.tar -o sys
#> gzip --suffix .vgz sys

Move the sys.vgz image back into the extracted iso image
#> mv sys.vgz /esxi41u1

And finally, generate a new bootable .iso image from our modified bits:
#> cd /esxi41u1
#> mkisofs -l -J -R -r -T -input-charset UTF-8 -o /custom_esxi4.1u1.iso -b isolinux.bin -c boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table /esxi41u1/


Unknown file type (unallocated) ./.. - ignoring and continuing.
Size of boot image is 4 sectors -> No emulation
2.49% done, estimate finish Sat Feb 26 02:19:12 2011
4.97% done, estimate finish Sat Feb 26 02:19:12 2011
7.44% done, estimate finish Sat Feb 26 02:19:12 2011
9.93% done, estimate finish Sat Feb 26 02:19:12 2011
12.41% done, estimate finish Sat Feb 26 02:19:12 2011
14.89% done, estimate finish Sat Feb 26 02:19:12 2011
17.37% done, estimate finish Sat Feb 26 02:19:17 2011
19.85% done, estimate finish Sat Feb 26 02:19:17 2011
22.33% done, estimate finish Sat Feb 26 02:19:16 2011
24.82% done, estimate finish Sat Feb 26 02:19:16 2011
27.29% done, estimate finish Sat Feb 26 02:19:15 2011
29.78% done, estimate finish Sat Feb 26 02:19:15 2011
32.25% done, estimate finish Sat Feb 26 02:19:15 2011
34.74% done, estimate finish Sat Feb 26 02:19:14 2011
37.22% done, estimate finish Sat Feb 26 02:19:14 2011
39.70% done, estimate finish Sat Feb 26 02:19:14 2011
42.18% done, estimate finish Sat Feb 26 02:19:16 2011
44.66% done, estimate finish Sat Feb 26 02:19:16 2011
47.14% done, estimate finish Sat Feb 26 02:19:16 2011
49.63% done, estimate finish Sat Feb 26 02:19:16 2011
52.10% done, estimate finish Sat Feb 26 02:19:15 2011
54.59% done, estimate finish Sat Feb 26 02:19:15 2011
57.06% done, estimate finish Sat Feb 26 02:19:15 2011
59.55% done, estimate finish Sat Feb 26 02:19:15 2011
62.03% done, estimate finish Sat Feb 26 02:19:15 2011
64.51% done, estimate finish Sat Feb 26 02:19:15 2011
66.99% done, estimate finish Sat Feb 26 02:19:14 2011
69.47% done, estimate finish Sat Feb 26 02:19:14 2011
71.95% done, estimate finish Sat Feb 26 02:19:16 2011
74.44% done, estimate finish Sat Feb 26 02:19:16 2011
76.91% done, estimate finish Sat Feb 26 02:19:15 2011
79.40% done, estimate finish Sat Feb 26 02:19:15 2011
81.88% done, estimate finish Sat Feb 26 02:19:15 2011
84.35% done, estimate finish Sat Feb 26 02:19:15 2011
86.84% done, estimate finish Sat Feb 26 02:19:15 2011
89.32% done, estimate finish Sat Feb 26 02:19:15 2011
91.80% done, estimate finish Sat Feb 26 02:19:16 2011
94.28% done, estimate finish Sat Feb 26 02:19:16 2011
96.76% done, estimate finish Sat Feb 26 02:19:16 2011
99.24% done, estimate finish Sat Feb 26 02:19:16 2011
Total translation table size: 2048
Total rockridge attributes bytes: 0
Total directory bytes: 2970
Path table size(bytes): 26
Max brk space used 0
201533 extents written (393 MB)

The finished .iso image will be named custom_esxi4.1u1.iso, and should be in your root directory.


(optional, if you want to pxe boot the image )
Create an image.tgz that can be used for PXE booting:
#> mkdir -p /home/vi-admin/usr/lib/vmware/installer
#> cp /esxi41u1/imagedd.bz2 /home/vi-admin/usr/lib/vmware/installer/VMware-VMvisor-big-4.1.0-348481-x86_64.dd.bz2
#> cd /home/vi-admin
#> tar cvzf image.tgz usr/
usr/
usr/lib/
usr/lib/vmware/
usr/lib/vmware/installer/
usr/lib/vmware/installer/VMware-VMvisor-big-4.1.0-348481-x86_64.dd.bz2

The resultant image.tgz can be used when PXE booting to do a manual installation of ESXi 4.1 U1, and will have the be2net driver loaded at installation time.

vibddi & Customized ESXi Drivers

Earlier today I had a shipment of HP blades come in (HP BL460 G7). I tried to PXE boot them with the ESXi 4.1 installer, but the installer failed as even the latest version of ESXi is missing a necessary driver ( a network adapter in my case ). Curious, I fired up the VMware Auto Deploy appliance, which contains a tool called vibddi which allows you to customize ESXi images. William Lam has written about it before, and I looked through his blog post and figured out how the tool works:

virtuallyGhetto: How to inject custom drivers into an ESXi image using vibddi


Trying to figure out exactly what was in the HP image, I first took a look at the stock ESXi 4.1 vibs:

[root@auto-deploy ~]# vibddi -i imagedd -q
vmware-esx-drivers-net-be2net (400.2.102.518.0-1vmw.2.17.249663)
vmware-esx-firmware (4.1.0-0.0.260247)
CMPI-1.0
VMW_CMPI_CPP-1.0.5
vmkapi_1_1_0_0
vmknexus1kvapi-0-0
vmkepsecapi_1.0.0.0
DriverAPI-9.0
DriverAPI-9.1
vmware-esx-tools-light (4.1.0-0.0.260247)
(sidenote: I found it interesting that the Nexus 1000v is baked into the default ESXi installer )

Then compared it to the set of drivers in the HP custom ESXi bundle:
[root@auto-deploy root]# vibddi -i imagedd -q
oem-hp-smx-provider (410.02.07.60-235786)
oem-hp-support (410.1.1.0-164009)
oem-vmware-esx-drivers-char-hpilo (400.8.5.0.1VMW-164009)
oem-vmware-esx-drivers-net-be2net (400.2.102.440.0-1vmw.2.17.249663)
oem-vmware-esx-drivers-net-nx-nic (400.4.0.560-1.0.4.00000)
vmware-esx-drivers-char-hpnmi (410.1.0.0.1VMW-235786)
vmware-esx-firmware (4.1.0-0.0.260247)
CMPI-1.0
VMW_CMPI_CPP-1.0.5
vmkapi_1_1_0_0
vmknexus1kvapi-0-0
vmkepsecapi_1.0.0.0
DriverAPI-9.0
DriverAPI-9.1
vmware-esx-tools-light (4.1.0-0.0.260247)

The resultant set shows these additional components in the HP specific VIBs:
oem-hp-smx-provider (410.02.07.60-235786)
oem-hp-support (410.1.1.0-164009)
oem-vmware-esx-drivers-char-hpilo (400.8.5.0.1VMW-164009)
oem-vmware-esx-drivers-net-be2net (400.2.102.440.0-1vmw.2.17.249663)
oem-vmware-esx-drivers-net-nx-nic (400.4.0.560-1.0.4.00000)
vmware-esx-drivers-char-hpnmi (410.1.0.0.1VMW-235786)

After looking at the driver I was missing, I found the offline VIB bits on the VMware website, injected it into the installer with vibddi, created a new image.tgz for PXE booting, and I was back in business. To be fair, I could have just set up the PXE boot server with the bits from the HP custom ESXi image, but I wanted to introduce as little variation into my environment as possible.

But wait, there's more! vibddi will let you inject basically anything into the installer that in VIB format, you can create custom install images that are already patched with the latest firmware/tools, inject 3rd party software packs like EMC's PowerPath PP/VE and considerably reduce the amount of time it takes to take a bare metal server from no image installed, to fully deployed. You can even inject a custom ks.cfg and create pxe bootable ( or CD ) images that are customized for each one of your systems.