Hanlon does Windows!

One of the most often requested features whenever we’ve talked with Hanlon users (going all the way back to when Nick Weaver and I released Hanlon under it’s original name, Razor) has been Windows support. We’ve struggled with how to add support for Windows provisioning to both Razor and Hanlon for a couple of years now, and we’ve even had a few false starts at providing support for the feature, but somehow the implementations we tried never really made it into a production Razor/Hanlon server.

The issue hasn’t been a technical one, instead it’s been an issue of how to fit the provisioning of Windows and the management of Windows images into the provisioning workflow used by Hanlon. Windows is, well how shall we put this, just a bit different from the other operating systems and hypervisors that we’ve supported to date and we have struggled all along to find a way to integrate the workflow used for the the unattended Windows install process with the workflow Hanlon uses to automate the install of the operating systems and/or hypervisors that we already support.

That being said, today we are formally announcing that Hanlon now provides fully-automated provisioning of Windows instances in your datacenter using a workflow that should be familiar to both Hanlon users and Windows administrators. There are still a few features that remain to be implemented (mainly around the when notification is sent back to Hanlon that a node is “OS Complete” and support for “broker handoffs” to Windows nodes), but we felt that it was better to get these features out in public so that we could get feedback (and pull requests?) from the community as soon as possible.

In this blog posting, I’d like to walk you through the new Windows support we’ve added to Hanlon. Along the way, I’ll talk through some of the features we had to add to Hanlon in order to support Windows provisioning and highlight some of the differences in workflow for those Windows administrators who have not used Hanlon before and those Hanlon users not familiar with how unattended Windows installs work. For those of you who don’t have the time or patience to read the rest of this blog posting, you can find a screencast of yours truly using these new features posted here. As always, feedback and comments are more than welcome, and if anyone would like to help us improve these features, please do let us know.

Bare-metal Provisioning of Windows

As I mentioned in my introductory remarks, the process of bare-metal provisioning Windows via Hanlon is slightly different from the process that Hanlon follows when provisioning a Linux-based OS or a VMware or Xen Hypervisor. This is due to differences between how the Windows ISO is structured when compared to the Linux or Hypervisor ISOs that Hanlon has supported to date and the differences between how an unattended Windows install works and how a Linux or Hypervisor automated install works. As a result, an unattended Windows install requires that a few external components must be setup and configured before Hanlon can successfully iPXE-boot a Windows-based OS install.

Hanlon’s new static ‘slice’

The first big difference for unattended Windows installs is that the iPXE-boot process for that installation process relies on the ability to download a number of components from a web server that is available in the iPXE-boot network. Rather than requiring that users setup (and configure) an external web server, we have decided to add a new static area to Hanlon itself. To make use of this new capability, simply add a hanlon_static_path parameter to your Hanlon server configuration that points to the directory you wish to use for serving up static content through Hanlon. Any content placed under the directory will then be available vi a GET operation against the /static RESTful endpoint.

With that new static area configured in your Hanlon server, the next step is to setup the appropriate structure under that area to support iPXE-booting of a server into WinPE via Hanlon. The tree structure that you are setting up should look something like this:

$ tree
├── boot
│   ├── bcd
│   └── boot.sdi
├── sources
│   └── boot.wim
└── wimboot

2 directories, 4 files

The files that are placed into this directory tree (the bcd, boot.sdi, boot.wim, and wimboot files) come from a variety of sources. The boot.wim file under the sources directory is the WinPE image you wish to use to boot your hardware. This WinPE image will have to be built separately, and will likely have to be customized to suit your hardware (for those interested, Joe Callen has put together a blog posting of his own that describes this process, you can find his posting here). Unfortunately, licensing restrictions don’t allow for redistribution of ‘pre-built’ WinPE images but, as you can see in Joe’s post, we’ve tried to make the process of building this image as simple as possible (even for non-Windows developers).

The boot/bcd and boot/boot.sdi files can probably be obtained from any Windows ISO, although the easiest location to grab them from is probably the ISO you are going to install Windows from. These files can be obtained by mounting a Windows ISO and copying them over or they can be copied out of the directory created when you add a Windows ISO to Hanlon (more on this later). When copying over these files, keep in mind that while Windows is not case-sensitive when it comes to filenames, the server you are running your Hanlon server likely is. As such, make sure that the filenames you create in the static area match the case of the corresponding file in the tree structure shown, above (the only real issue is is probably the boot/bcd file, which will appear as boot/BCD on a Windows ISO).

Lastly, the wimboot file can be obtained from any recent build of the wimboot project (the latest version is typically available here, but older versions can be found through the project’s GitHub repository, which can be found here). Once these files are in place, your Hanlon server’s static area is ready to be used.

Changes to the DHCP server configuration

Since the DHCP client used by WinPE does not support passing of DHCP options in the same way as that of the DHCP client used in most Linux/Hypervisor distributions, some minor changes to your DHCP server configuration are probably necessary. Specifically, the section of your DHCP server that looks like this:

# specify a few server-defined DHCP options
option hanlon_server code 224 = ip-address;
option hanlon_port code 225 = unsigned integer 16;
option hanlon_base_uri code 226 = text;

will have to be modified to support both Linux and Windows PE clients as follows:

# specify a few server-defined DHCP options
option hanlon_server code 224 = ip-address;
option hanlon_port code 225 = unsigned integer 16;
option hanlon_base_uri code 226 = text;

# options used for Windows provisioning
option space hanlon;
option hanlon.server code 224 = ip-address;
option hanlon.port code 225 = unsigned integer 16;
option hanlon.base_uri code 226 = text;

Note that in the case of Windows PE clients, we rely on a hanlon space to pass through the parameters that the Windows PE client will need to successfully connect back to the Hanlon server and retrieve the active_model parameters that it needs to continue with the appropriate Windows install (based on the model that it was bound to).

Without these additional runtime parameters, we would have to customize our Windows PE image so that it knew how to contact Hanlon in order to retrieve the active_model instance that it has been bound to (which contains information needed by the WinPE instance to perform an unattended Windows install). With these additional parameters, it is actually quite simple to put together a generic Windows PE image that can connect back to the Hanlon server (via a simple PowerShell script) to obtain this information.

To finish off the task of reconfiguring your DHCP server, you’ll also have to make use of the new space that was defined, above. To accomplish this, simply track down the lines that look like this in your current DHCP server configuration file:

  option hanlon_server;
  option hanlon_port 8026;
  option hanlon_base_uri "/hanlon/api/v1";

and modify that section of your DHCP server configuration file so that it looks like this instead:

  class "MSFT" {
    match if substring (option vendor-class-identifier, 0, 4) = "MSFT";
    option hanlon.server;
    option hanlon.port 8026;
    option hanlon.base_uri "/hanlon/api/v1";
    vendor-option-space hanlon;
  class "OTHER" {
    match if substring (option vendor-class-identifier, 0, 4) != "MSFT";
    option hanlon_server;
    option hanlon_port 8026;
    option hanlon_base_uri "/hanlon/api/v1";

With those changes in place, your DHCP server should now be ready to support chain booting of your machines into a Hanlon-based Windows install.

Building a WinPE image that supports your hardware

In order to make the process of provisioning Windows as painless as possible, Joe Callen (@jcpowermac) created a simple PowerShell script that can be used (on a Windows machine) to build a WinPE image that is suitable for use with your hardware. Drivers for specific networking and storage devices are typically needed to successfully iPXE-boot a node and install an OS instance on bare-metal hardware, and in the case of Windows the drivers for these networking and storage devices must be included as part of the WinPE image that is used to drive the installation process.

Do keep in mind, however, that our goal here is to make the resulting WinPE image as generic as possible so that it can be easily reused with all of your hardware. Rather than embedding a lot of model-specific logic into the WinPE image, we’ve worked quite hard to define a few custom PowerShell scripts that can be used to download appropriate versions of the key files/scripts directly from Hanlon (with appropriate values filled in based on the active_model instance that was bound to the current node). I’ll avoid going into the specifics here; Joe has provided a much more complete discussion of this process in his recent blog post, which is available here.

Loading a Windows ISO into Hanlon

As was mentioned earlier, the structure of a Windows ISO is quite different from the structure of the typical Linux/Hypervisor ISO. The biggest difference is that while the typical Linux/Hypervisor ISO contains a single image, a Windows ISO actually contains a number of different Windows images packaged up into a single install.wim file. This difference in structure shows up in the way that an automated install of Windows actually works. As part of the “autounattend” file you present to the Windows installer, you have to specify not only the location of the install.wim file to use for the install but also the WIM Index of the image in that file that you wish to use for the install.

So what does all of this mean for our image slice under Hanlon? To put it quite simply, we’ve had to make a significant change to how Hanlon handles images in order to support loading of Windows ISOs by the image slice. Specifically, the process of adding a single Windows ISO to Hanlon has the effect of creating multiple (linked) Hanlon images, a concept that was not needed for any of the Linux/Hypervisor ISOs that were already supported by Hanlon.

In order to minimize the space that the ‘unpacked’ ISO takes on disk, we maintained the requirement that each ISO would create a single directory on disk (containing the contents of that ISO). What had to change to support Windows was to add the concept of multiple, linked image objects in Hanlon (each of which is linked to a single base image that actually maps to the underlying directory created during the image add process). This base image then becomes a ‘hidden’ image (one that cannot be seen or used), and it is this base image that is actually used by any of the linked images in order to access resources that are associated with these images (like the install.wim file).

To facilitate this new concept, we modified the Hanlon image object type to take advantage of a ‘hidden’ field that was already defined for all Hanlon objects. When a Windows ISO is imported to Hanlon, the following sequence of operations are performed:

  • The ISO is unpacked into a locally accessible directory and a Windows ‘base image’ is created. This image is ‘hidden’ and, as such, will not appear in the list of Hanlon images returned by a GET against the /image endpoint by default
  • A standard Linux utility (the wiminfo utility) is then used to parse the install.wim file associated with that base image. It should be noted that Hanlon will look for this install.wim file under the sources subdirectory for any given base image (eg. under the the ${IMAGE_PATH}/windows/6hQHOAhWyuiLpUmuZlfQAa/sources directory for a base image with the uuid of 6hQHOAhWyuiLpUmuZlfQAa)
  • The information returned from the wiminfo command is then used to create a set of image objects that are linked to the underlying base image that was created when the ISO was unpacked. These linked images are only valid image objects so long as the underlying base image remains intact. Each of these linked images correspond to one of the images found when the underlying install.wim file from the base image was parsed using the wiminfo command.

The result of adding a single Windows ISO to Hanlon as an image (using the hanlon image add command or it’s RESTful equivalent) will be something like the following:

$ hanlon image add -t win -p /tmp/win2012r2.iso
Attempting to add, please wait...
         UUID                Type                               Name/Filename                          Status
2GLbzghe1VABDmYVtlLQe0  Windows Install  Windows Server 2012 R2 Standard (Server Core Installation)    Valid
2GLcLzQbh6Uxid9Nei84cy  Windows Install  Windows Server 2012 R2 Standard (Server with a GUI)           Valid
2GLccHsh97YE5BA9H0ghqi  Windows Install  Windows Server 2012 R2 Datacenter (Server Core Installation)  Valid
2GLcp9biBf0uMz80EwGBmK  Windows Install  Windows Server 2012 R2 Datacenter (Server with a GUI)         Valid
$  hanlon image 2GLbzghe1VABDmYVtlLQe0
 UUID =>  2GLbzghe1VABDmYVtlLQe0
 Type =>  Windows Install
 Name/Filename =>  Windows Server 2012 R2 Standard (Server Core Installation)
 Status =>  Valid
 OS Name =>  Windows Server 2012 R2 Standard (Server Core Installation)
 WIM Index =>  1
 Base Image =>  eeQtUH3Id2xs96i0Cft7w

As you can see from this example, the ISO was unpacked into a base image (with a UUID of eeQtUH3Id2xs96i0Cft7w) and four linked images (with the UUIDs shown in the output of the hanlon image add command shown above). While the linked images are independent of each other, all of them depend on the underlying base image for their ‘contents’. As such, removing the filesystem associated with the underlying base image will render all four (in the above example) of these linked images invalid.

Methods used to ‘unpack’ Windows ISOs

It should be noted here that the process for unpacking Windows also may differ from that used to unpack Linux or Hypervisor ISOs. In the Linux/Hypervisor case, fuseiso can be used to ‘mount’ the ISO if it is available. If the fuseiso command cannot be found then a regular mount command is attempted as a fallback. If both of those commands fail, then an error is thrown indicating that the user cannot add the image to Hanlon. If either of those commands succeed, then the contents of the ISO are copied over from the mount-point to a directory on the Hanlon filesystem that is under the ${IMAGE_PATH} directory.

Unfortunately, in the case of Windows, the fuseiso command cannot be used to mount a Windows ISO since the UDF (Universal Data Format) file system format used with Windows ISOs is not supported by the fuseiso command (even though it does support the ISO 9600 and Joliet formats). While the mount command could still be used to mount and copy over the contents of a Windows ISO, we didn’t want to add a requirement to Hanlon that the Hanlon server be run as a user with sudo rights to execute the mount/unmount commands.

To get around this limitation of the fuseiso command, we have added support to Hanlon for use of the 7z command for unpacking of a Windows ISO into a subdirectory of the ${IMAGE_PATH} directory. 7z was chosen as an alternative because, even though it cannot properly read the Joliet filesystem used with some Hypervisor ISOs (specifically ESX 5.x ISOs), it does support the UDF filesystem used with Windows ISOs. The biggest difference of using 7z versus the previous methods supported is that the ISO is never mounted on the Hanlon filesystem, instead the contents of the ISO are directly extracted to the target directory.

A note on removing Windows images

As you can see, this single hanlon image add command created a set of 4 linked images. The underlying base image is not visible in this view, nor is it visible in any of the default views provided by Hanlon. To see the underlying base image details, we actually need request the base image details specifically (using a command like hanlon image eeQtUH3Id2xs96i0Cft7w in the example shown above) or we need to make use of the new --hidden flag that we’ve added to the hanlon image command to facilitate the display of all of the images currently available, including the hidden ones:

$ hanlon image --hidden
         UUID                Type                               Name/Filename                          Status
2GLbzghe1VABDmYVtlLQe0  Windows Install  Windows Server 2012 R2 Standard (Server Core Installation)    Valid
2GLcLzQbh6Uxid9Nei84cy  Windows Install  Windows Server 2012 R2 Standard (Server with a GUI)           Valid
2GLccHsh97YE5BA9H0ghqi  Windows Install  Windows Server 2012 R2 Datacenter (Server Core Installation)  Valid
eeQtUH3Id2xs96i0Cft7w   Windows Install  Windows (Base Image)                                          Valid
2GLcp9biBf0uMz80EwGBmK  Windows Install  Windows Server 2012 R2 Datacenter (Server with a GUI)         Valid

So, given that we now have images in Hanlon that are linked together, how do we handle removal of these images? Previously, Hanlon just removed the underlying directory containing the unpacked version of the ISO that was used create the image, then removed the corresponding image object from Hanlon. There is also a check to ensure that the image you are removing is not a part of a model that is currently defined in Hanlon. If it is part of a model, then removal of the underlying image is prohibited (to prevent removal of an image from breaking any existing models defined in Hanlon that might be using that image).

Hopefully the new rules for removing Windows images are fairly apparent (the rules for removal of Microkernel, Linux or Hypervisor images remain unchanged), but in case they are not, here’s a short summary:

  • if an image is a base image, removal of that image will remove all of the images that are linked to that image, the base image itself, and the directory containing the contents of the ISO that was created when that base image was added to Hanlon
  • if an image is a linked image, then removal of that image will only result in removal of the linked image object itself; after that linked image is removed, if there are no other images remaining that are linked to the base image of that linked image, then the underlying base image (and the directory containing the contents of the ISO that was created when that base image was added to Hanlon) will also be removed
  • requests to remove a linked image will be blocked if the image in question is used in a model currently defined in Hanlon
  • requests to remove a base image will be blocked if any of the images that link to that base image are used in a model currently defined in Hanlon

Creating a Windows model

Once a Windows ISO has been loaded, it is relatively simple to use one of the images created by that process to create a Windows model. The command to add a new Windows model to Hanlon will look something like the following:

$ hanlon model add -t windows_2012_r2 -l windows_2012_r2_dc -i 2GLcc
--- Building Model (windows_2012_r2):

Please enter Windows License Key (example: AAAAA-BBBBB-CCCCC-DDDDD-EEEEE)
(QUIT to cancel)
Please enter node hostname prefix (will append node number) (example: node)
default: node
(QUIT to cancel)
Please enter local domain name (will be used in /etc/hosts file) (example: example.com)
default: localdomain
(QUIT to cancel)
Please enter admin password (> 8 characters) (example: P@ssword!)
default: test1234
(QUIT to cancel)
Please enter User Name (not blank) (example: My Full Name)
default: Windows User
(QUIT to cancel)
Please enter Organization (not blank) (example: My Organization Name)
default: Windows Organization
(QUIT to cancel)
Model Created:
 Label =>  windows_2012_r2_dc
 Template =>  windows_deploy
 Description =>  Windows 2012 R2
 Image UUID =>  2GLccHsh97YE5BA9H0ghqi


As you can see, using one of these linked Windows images is exactly the same as using a Linux or Hypervisor image; the only differences from creating a Linux model is the template name are the additional Windows License Key, User Name, and Organization fields that must be entered for a Windows model.

Creating a Windows policy

Creating a Windows policy is even simpler. The arguments for a Windows policy are exactly the same as those for a Linux or Hypervisor deployment policy (except, of course for the use of the windows_deploy template in the hanlon policy add ... command):

$ hanlon policy add -p windows_deploy -t 'ebig_disk,memsize_2GiB' -l windows_2012_r2_dc -m 3xFEK -e true
Policy Created:
 UUID =>  31GY0H7Dohh7hkjhGhDXXs
 Line Number =>  10
 Label =>  windows_2012_r2_dc
 Enabled =>  true
 Template =>  windows_deploy
 Description =>  Policy for deploying a Windows operating system.
 Tags =>  [ebig_disk, memsize_2GiB]
 Model Label =>  windows_2012_r2_dc
 Broker Target =>  none
 Currently Bound =>  0
 Maximum Bound =>  0
 Bound Counter =>  0


As you can see, the result is a Windows deployment policy that can be used to bind the underlying Windows model to a node that matches this policy (based on the tags assigned to that node).

Booting your Windows machine

Once you’ve loaded a Windows ISO into Hanlon as a set of linked Windows images and you’ve created a model and policy based on one of those linked images, the rest is simple and works exactly like it does for the Linux/Hypervisor installs you have already done. Simply configure a node that will be matched to your Windows model by your Windows policy so that it will network boot on the Hanlon server’s network, then power it on. The node will then chain boot (from a PXE-boot via TFTP to an iPXE-boot via Hanlon) and Hanlon will send back an iPXE-boot script that will trigger a WinPE-based unattended Windows install.

The workflow may be a bit different — requiring an extra reboot versus what you’re used to with the Linux/Hypervisor installs you’ve already done with Hanlon to date — but the process internally remains the same. The WinPE image reaches back to Hanlon via a RESTful request and obtains a PowerShell script that it uses to automate the process of setting up an unattended Windows install. In that script, it downloads the appropriate install.wim file from Hanlon along with an autounattend.xml file that will be used to control the unattended Windows install process. The autounattend.xml file that it downloads will be customized based on the specific Hanlon model that the node in question was bound to, and will contain all of the details that are needed to complete the unattended Windows install (the location of the install.wim file, the WIM Index of the image in that install.wim file that should be installed, the Windows license key, the hostname, the Administrator password, etc.). Finally, that script will also download a set of drivers from Hanlon and inject them into the downloaded install.wim. Currently, these drivers are assumed to be packaged in a single drivers.zip file that is available at the root of the static area that we added to Hanlon in our most recent release, but down the line we may end up extending how Hanlon downloads and injects these drivers in order to make this part of the process a bit easier to configure and extend.

With these tasks complete, the script then starts the standard setup process that will manage the unattended Windows install. When the unattended install is complete, you’ll have a fully-functional Windows instance that has been configured to match the parameters from the underlying Hanlon model that it was bound to.

In conclusion…

As always, we hope you all find this new set of features in Hanlon useful in your day-to-day work. We welcome feedback on these new features from anyone in the Hanlon or Windows communities, and look forward to your contributions as well. There are features we haven’t implemented yet (we still haven’t sorted out the process for handing off these Windows nodes to a Hanlon broker, for example), but we thought that even in this early stage of the game the features we’ve added to Hanlon are significant enough that we should release them into the wild, so to speak.

I’d also like to take this opportunity to thank Joe Callen (@jcpowermac) specifically. Without his unending support on the Windows side of this process (and patience with an old Linux/Unix dweeb like me), the seamless interaction between Hanlon and WinPE shown in these changes wouldn’t be nearly so seamless. Joe has worked long and hard on this set of changes, and deserves a great deal of the credit for where we are today.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s