Dell XR4000 – Witness Node Upgrade
If you run a two-node vSAN cluster at the edge, you need a witness. Dell’s PowerEdge XR4000w is a compact witness sled that does the job nicely, but it comes with a catch: it is a stripped-down server without an iDRAC, so you cannot simply mount an ISO through a virtual console. The first ESXi installation has to be done over a serial connection.
In this post I walk through the whole procedure we used to bring up a witness host in our data center: preparing the laptop, building a bootable USB stick with the right kernel options, installing ESXi 8.0 Update 3 over the serial console, configuring the management network, and finally installing Dell’s witness drivers so that the host shows up in the iDRAC of the vSAN nodes.
What you need
- A Windows laptop with a free USB-A port and admin rights
- A USB-A to micro-USB cable (the micro-USB port on the XR4000w front panel is the serial console)
- A USB stick (we used a 32 GB stick)
- The CP210x Universal Windows Driver (Silicon Labs USB-to-UART bridge)
- A terminal program such as PuTTY
- Rufus, to write the bootable USB stick
- The Dell-customized VMware ESXi 8.0 ISO, for example VMware-VMvisor-Installer-8.0.0.update03-24859861.x86_64-Dell_Customized-A08.iso
- The two Dell witness drivers for ESXi 8.0: Dell OS Agent for Witness and Dell OS Driver for Witness
About the Hardware
The XR4000w is a sled, built for the Dell PowerEdge XR4000 chassis. On the front you will find a power button, a USB port, two RJ-45 network ports and the micro-USB console port. There is no iDRAC and no video output, which is exactly why the serial console matters. The sled we installed reports an Intel Atom C3500 CPU at 1.60 GHz with 16 GiB of memory and a 960 GB NVMe M.2 boot device.
Step-by-Step Installation
1. Install the serial driver on your Notebook
To open a serial connection over USB, Windows needs the CP210x Universal Windows Driver. Download and install it, then open Device Manager as administrator (devmgmt.msc). If the adapter shows up with a yellow “!”, right-click it and update the driver.
2. Connect the Notebook to the Witness Sled
Plug the USB-A end of the cable into your laptop and the micro-USB end into the console port on the front of the XR4000w. Windows should now assign a new COM port. Check Device Manager to see which one it is (in our case COM4).
3. Open the serial session in PuTTY
In PuTTY, choose the connection type Serial and enter the following:
Serial line: COM4 (use the port from Device Manager)
Speed: 115200
Once connected, you should start seeing output from the witness host.
4. Create the bootable USB stick
- Download the Dell-customized ESXi 8.0 ISO. It includes the Dell drivers the XR4000w needs.
- Open Rufus, select your USB stick and the ISO, and write the image. We used the defaults: MBR partition scheme, target system “BIOS or UEFI” and FAT32.
5. Edit boot.cfg to redirect the console to the serial port
This is the key trick. Without it the ESXi installer will not talk to your PuTTY session. Open BOOT.CFG on the USB stick and append the following values to the existing kernelopt line. Do not overwrite the parameters that are already there.
text nofb com2_baud=115200 com2_Port=0xe030 tty3Port=com2
Tip: These options send the ESXi console to the correct hardware serial device and to the micro-USB port on the XR4000w. A typo here usually means a blank PuTTY window, so double-check the line before you boot.
6. Boot the witness host from the USB stick
Insert the stick into the USB port on the front of the sled and power it on. In PuTTY you will see the ESXi boot messages scroll by: services such as DCUI, esxui, vSAN components and various Dell services are started one after another.
7. Run the ESXi Installer
When the boot has finished, the VMware ESXi 8.0.3 installer appears in the serial console. Walk through it as usual:
- Confirm the welcome screen to start the installation.
- Select the target disk. In our case this was the 960 GB Dell NVMe M.2 device. Be careful not to pick the USB stick, which is also listed (about 29 GiB in our case).
- Confirm the installation. The installer warns that the disk will be repartitioned. Press F11 to install.
- When the installation completes, remove the USB stick and reboot.
After the reboot, the console shows the familiar ESXi start screen with the product version (VMkernel Release Build 24859861), the hardware model (Dell Inc. PowerEdge XR4000w) and the management URL.
8. Configure the management network
Press F2 to open “Customize System” and go to Configure Management Network. The relevant menus are:
- Network Adapters: check that the right vmnic is selected (the port you have cabled).
- VLAN (optional): set a VLAN ID if your management network is tagged, otherwise leave it unset.
- IPv4 Configuration: switch to a static address and enter the IP address, subnet mask and default gateway.
- IPv6 Configuration: disable IPv6 if you do not use it.
- DNS Configuration: enter the primary and alternate DNS servers and the hostname.
- Custom DNS Suffixes: add your domain so short names resolve.
Apply the changes and let the management network restart. From here on you can reach the host with a browser or via SSH, if you enable it.
9. Install the Dell witness drivers
The last piece is optional in the sense that ESXi will run without it, but we strongly recommend it. Dell provides two driver packages for the witness, which you can download from the XR4000w support page by selecting VMware ESXi 8.0 as the operating system:
- Dell OS Agent for Witness
- Dell OS Driver for Witness
They are valuable because they provide a heartbeat function that reboots a locked XR4000w, and they pass the ESXi host’s IP information to another XR4000w sled, via the iDRAC user interface of a Dell PowerEdge XR4510c or XR4520c server in the same chassis.
Copy both ZIP files to the host (we stored them on an internal software share and then uploaded them to /var/log/vmware), enable the ESXi shell or SSH, and install the two packages:
esxcli software vib install -d /var/log/vmware/Dell-ACPI-GPIO-Driver-Web-1.2.0.0-70.VIB-ESX8i.zip
esxcli software vib install -d /var/log/vmware/DOSA-Witness-Web-1.2.0.0-70.VIB-ESX8i.zip
Each command reports that the update completed successfully but that a reboot is required for the change to take effect. That is expected.
Next, edit the boot configuration and adjust the line as described in Dell’s documentation:
vi /bootbank/boot.cfg
Reboot the host. In our experience it took several reboots until everything was picked up correctly, so do not be alarmed if the first restart does not show the result yet.
Checking the result
Now for the satisfying part. Log in to the iDRAC of the vSAN hosts that sit in the same chassis and open System → Overview → Witness Server. You should see the witness sled with its status, power status, power consumption, service tag, and – most importantly – the IP and MAC addresses of the witness host, plus the heartbeat state.
If the IP address appears there, the driver and agent are working, the heartbeat is active, and the witness is ready to be added to your vSAN cluster as a witness host.
Leassons learned and tips
- Get the kernel options right. Almost every “black screen” problem comes from the boot.cfg line. Append, never replace.
- Check the COM port every time. If you use a different laptop or USB port, Windows may assign a different number.
- Pick the correct disk. The installer lists the USB stick next to the NVMe device. Choose the NVMe.
- Keep the drivers on a shared location. We keep them in our software repository so every future witness can be built the same way.
- Plan for multiple reboots after installing the Dell VIBs and editing boot.cfg.
That’s it from this Blog Post, if you have any questions please use the comment section below or contact Dell Support! This blog post wouldn’t have been possible without my friends at KVH; Daniel Zinser and Christian Zinke – they played a major role in it – Thank you Guys! 🙂