What Determines the Protocol Used During a VMware VM PXE Boot Against a WDS Server?

Good-day folks,

I have successfully implemented a Windows Deployment Server in my environment, built both x86 and x64 boot images to support UEFI and Legacy BIOS using the Microsoft Deployment Toolkit (v8450), and have been able to successfully PXE-Boot both a physical machine (HP Z4) and a Virtual Machine (VM version 13 running on ESXi 6.5 host). Additionally, I have created the relevant DHCP Vendor Classes and Policies to ensure the right boot file is being served to the appropriate PXEClient (i.e., UEFI vs BIOS).

That said, when booting using UEFI (with Secure Boot enabled), I have observed two things (the second being of concern, hence my posting this question):

1. During the boot process, the physical machine successfully contacts the WDS Server and pulls down the boot image in about 6 seconds, then goes on to complete the OS install in about 6 – 8 minutes.

2. During the boot process, the virtual machine also successfully contacts the WDS Server and pulls down the boot image. However, it takes upwards of 45 minutes to 1 hour to grab the boot image. After that, the OS install completes in about the same time as the physical box (just about 8 minutes).

To troubleshoot, I confirmed that the WDS Server wasn’t experiencing any technical issues that would affect its NIC performance. I did check the VM’s NIC configuration and found it to be using the E1000E adapter type – so I changed it to use the VMXNET 3 adapter type instead. This made a slight improvement.

So, I decided to do a Wireshark packet capture to see if I could see anything different in the way the physical machine and the VM were communicating with the WDS Server. Well the only thing that stood out to me is the following:

1. The Physical Machine used the TFTP protocol
2. The Virtual Machine used the UDP protocol

As such, does someone know if there’s a setting somewhere that’s instructing the Virtual Machine to prefer the UDP protocol over TFTP? I have found nothing definitive in the VMware documentation other than a statement to the fact that PXE-Booting a VM was possible and supported.

Any assistance is definitely appreciated, thanks and my apologies for the lengthy narrative.

So for anyone experiencing this issue, the solution for me was to adjust the “ramdisktftpblocksize” and the “ramdisktftpwindowsize” settings within the Boot Configuration Data (BCD) file for the respective architecture (i.e., x86 or x64) in the WDS RemoteInstall Boot folder. After which I restarted the WDS Service. I am now seeing about 10-15 seconds boot image load time on Virtual Machines and about 6-10 seconds on Physical Machines using a roughly 400MB boot image.

Only thing I noticed is that this creates fragmented IP packets, but they are re-assembled anyway, so doesn’t bother me much since there will be very little impact to my end users or network anyway based on the frequency and times at which I will be PXE-Booting systems for imaging. However, if that’s a problem for you then seek another solution please.

In my environment, the paths were as follows (yours might be different of course, so adjust accordingly):

For 32-bit (x86):  "F:SCCMRemoteInstallBootx86default.bcd"
For 64-bit (x64):  "F:SCCMRemoteInstallBootx64default.bcd"

CAUTION: Make a backup of the default.bcd file before you make changes, for obvious reasons (I hope).

Here’s what I did to implement this solution:

1. Open the WDS Management Console on the PXE Server (in my case it was the SCCM Server running the Windows Deployment Services role) and STOP the server. You could also just stop the WDS Service – but I preferred the former (as it’s a cleaner approach IMHO).
2. Launch an Administrative Command Prompt window (don’t be tempted to use a PowerShell console, I didn’t have much luck interacting with the BCDEdit command).
3. In a separate Windows Explorer window, locate the appropriate “default.bcd” file and either take note of its path or simply copy it into the clipboard by “SHIFT+Right-Click” and “Copy as path”.
4. With the WDS Server stopped and the path to the default.bcd file in hand, go back to the Command Prompt and type the following command (substituting x86 or x64 as appropriate):
   • List the current boot store parameters with this command:
     • bcdedit /enum all /store F:SCCMRemoteInstallBootx86default.bcd
   • Set the TFTP Windows Size to a value of 8:
     • bcdedit /store F:SCCMRemoteInstallBootx86default.bcd /set {68d9e51c-a129-4ee1-9725-2ab00a957daf} ramdisktftpwindowsize 8
   • Set the TFTP Block Size to a value of 16384:
     • bcdedit /store F:SCCMRemoteInstallBootx86default.bcd /set {68d9e51c-a129-4ee1-9725-2ab00a957daf} ramdisktftpblocksize 16384
   • List the current boot store parameters again and verify your changes are there:
     • bcdedit /enum all /store F:SCCMRemoteInstallBootx86default.bcd
5. Go back to the WDS Management Console and START the WDS Server

6. Signal the WDS Server to rebuild the boot files with your changes
   using the following command:**

   - sc control wdsserver 129
   

And that’s it. With these changes, I was able to fire up Wireshark on the PXE Server, pxe-booted my Virtual Machine. As I watched in Wireshark, I saw the initial conversations start out with “TFTPWindowSize = 4” and “TFTPBlockSize = 1456” but soon after when the boot loader itself started downloading I saw a renegotiation where the values I set took effect – and that’s when I saw the VM load the boot image within 10-15 seconds.

As I mentioned earlier, I saw about 8 or so “Fragmented IP protocol” packets but they were all reassembled and besides the end goal was achieve – the Virtual Machine and Physical Machine both booted up fast and installed the OS successfully.

Hope this information is helpful to someone – sorry for writing a book…lol.

Credits to the following post:
https://blog.uvm.edu/jgm/2010/11/04/tuning-microsoft-pxe-tftp/