Deploying OCI Container Images to Bare Metal with a Custom IPA Hardware Manager

What if you could deploy any OCI container image directly to bare metal, without building traditional disk images? Back in 2021, Dmitry Tantsur implemented custom deploy steps for Ironic, enabling alternative deployment methods beyond the standard image-based approach. This feature powers OpenShift’s bare metal provisioning with CoreOS, yet it remains surprisingly unknown to the broader Metal3 community. This post aims to change that by providing an example implementation of a custom IPA hardware manager that deploys Debian-based container images with EFI boot, LVM root filesystem, and optional RAID1 mirroring.

The Problem with Traditional Image-Based Deployments

Traditional bare metal provisioning with Metal3 and Ironic typically requires pre-built disk images. You need to maintain these images, update them regularly, and ensure they contain all necessary drivers and configurations. This approach has some drawbacks:

1. Image building complexity - Building and maintaining OS disk images is not as trivial as creating container images
2. Software RAID limitations - Image-based deployments with mdadm RAID and EFI boot require workarounds

What if we could leverage the container ecosystem instead? Container registries already solve the distribution problem, and OCI images are versioned, layered, simple to build and widely available. This approach allows you to:

• Use standard container images from any registry
• Avoid maintaining custom disk images
• Easily switch between OS versions by updating spec.image.url
• Get RAID1 redundancy with minimal configuration

Introducing the deb_oci_efi_lvm Hardware Manager

The DebOCIEFILVMHardwareManager is a custom IPA hardware manager that deploys Debian-based OCI container images directly to bare metal. It provides:

• EFI boot support - UEFI boot with GRUB, which unlike systemd-boot, supports booting from LVM on top of mdadm software RAID
• LVM root filesystem - Flexible volume management for the root partition
• Optional RAID1 - Software mirroring across two disks for redundancy
• Cloud-init integration - Ironic configdrive data is written directly to the root filesystem, no separate configdrive partition
• Multi-architecture - Supports x86_64 and ARM64 via OCI multi-arch images

How It Works

The deployment process extracts an OCI image using Google’s crane tool, then installs the necessary boot infrastructure on top. The hardware manager supports three methods for specifying the OCI image (in priority order):

1. spec.image.url with oci:// prefix (e.g., oci://debian:12)
2. Configdrive metadata annotation bmh.metal3.io/oci_image
3. Default fallback: ubuntu:24.04

Root device hints can be specified using either standard BareMetalHost rootDeviceHints fields or a simplified format via the bmh.metal3.io/root_device_hints annotation (e.g., serial=ABC123 or wwn=0x123456). For RAID1 configurations, provide two space-separated values (e.g., serial=ABC123 DEF456).

Note: Alternatively, podman can be used instead of crane for OCI image extraction, as it is readily available in CentOS Stream 9 and also has an export command. This would require code modifications to the hardware manager.

The hardware manager performs these steps during deployment:

1. Resolve OCI image - Check image_source, configdrive, or use default
2. Resolve target disks - Parse root device hints (serial or WWN)
3. Clean existing data - Wipe partitions, RAID arrays, and LVM based on disk wipe mode (all for RAID1, target for single disk by default)
4. Partition disks - Create 2GB EFI partition and LVM partition (with RAID1 if two disks are specified)
5. Create filesystems - FAT32 for EFI, ext4 for root LV
6. Extract OCI image - Use crane export piped to tar for rootfs
7. Install packages - Add cloud-init, GRUB, kernel, mdadm, lvm2
8. Configure boot - Set up GRUB, initramfs, and fstab
9. Install bootloader - GRUB to both EFI partitions for RAID1

Disk Layout

The hardware manager creates the following partition layout:

The LVM configuration:

For RAID1 configurations, both disks get identical partition tables, with partition 2 forming a RAID1 array that serves as the LVM physical volume.

Configuration

Basic Single-Disk Deployment

For a simple single-disk deployment, configure your BareMetalHost and Metal3MachineTemplate as follows:

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: my-server
  namespace: metal3
spec:
  online: true
  bootMode: UEFI
  # Preferred method: Use spec.image.url with oci:// prefix
  image:
    url: "oci://debian:12"
  rootDeviceHints:
    serialNumber: "DISK_SERIAL_NUMBER"

Alternatively, you can use annotations or simplified hint formats:

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: my-server-alt
  namespace: metal3
  annotations:
    # Alternative: Override default ubuntu:24.04 via annotation
    bmh.metal3.io/oci_image: "debian:12"
    # Alternative: Use simplified hint format
    bmh.metal3.io/root_device_hints: "serial=DISK_SERIAL_NUMBER"
spec:
  online: true
  bootMode: UEFI

The hardware manager supports three methods for specifying the OCI image (in priority order):

1. spec.image.url with oci:// prefix (highest priority, recommended)
2. Annotation bmh.metal3.io/oci_image passed via Metal3DataTemplate
3. Default ubuntu:24.04 (fallback)

Root device hints support both standard format (serialNumber: "ABC123") and simplified format via annotation (bmh.metal3.io/root_device_hints: "serial=ABC123").

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: my-worker-template
  namespace: metal3
spec:
  template:
    spec:
      customDeploy:
        method: "deb_oci_efi_lvm"
      dataTemplate:
        name: my-data-template

RAID1 Configuration

For production deployments requiring disk redundancy, specify two disk serial numbers. The hardware manager supports multiple formats:

Method 1: Standard format with space-separated values

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: my-ha-server
  namespace: metal3
spec:
  online: true
  bootMode: UEFI
  image:
    url: "oci://debian:13"
  rootDeviceHints:
    # Two space-separated serial numbers enable RAID1
    serialNumber: "DISK1_SERIAL DISK2_SERIAL"

Method 2: Simplified format via annotation

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: my-ha-server-alt
  namespace: metal3
  annotations:
    bmh.metal3.io/oci_image: "debian:13"
    # Simplified RAID1 hint format
    bmh.metal3.io/root_device_hints: "serial=DISK1_SERIAL DISK2_SERIAL"
spec:
  online: true
  bootMode: UEFI

With RAID1 enabled, the hardware manager will:

• Clean both disks (remove existing partitions, RAID arrays, and LVM)
• Create identical partition layouts on both disks
• Set up a RAID1 array (/dev/md0) for the LVM physical volume
• Install GRUB to both EFI partitions
• Configure a GRUB update hook to sync EFI partitions via rsync

Disk Wipe Mode Configuration

By default, the hardware manager wipes all block devices for RAID1 configurations (to prevent stray RAID/LVM metadata issues) and only target disks for single-disk setups. You can override this behavior:

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: my-server
  namespace: metal3
  annotations:
    # Control disk cleaning behavior
    # "all" - Wipe all block devices (recommended for RAID1)
    # "target" - Wipe only target disk(s) from root device hints
    bmh.metal3.io/disk_wipe_mode: "all"
spec:
  online: true
  bootMode: UEFI
  image:
    url: "oci://ubuntu:24.04"
  rootDeviceHints:
    serialNumber: "DISK_SERIAL_NUMBER"

The disk_wipe_mode annotation is useful when:

• You have multiple disks and want to ensure clean RAID/LVM state (all)
• You want to preserve data on non-target disks (target)
• You’re migrating from a previous RAID configuration

Metal3DataTemplate Configuration

When using annotations (instead of spec.image.url), configure your Metal3DataTemplate to pass them to the configdrive:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
  name: my-data-template
  namespace: metal3
spec:
  clusterName: my-cluster
  metaData:
    fromAnnotations:
    # Optional: Pass OCI image annotation (only if not using spec.image.url)
    - key: oci_image
      object: baremetalhost
      annotation: "bmh.metal3.io/oci_image"
    # Optional: Pass simplified root device hint
    - key: root_device_hints
      object: baremetalhost
      annotation: "bmh.metal3.io/root_device_hints"
    # Optional: Pass disk wipe mode
    - key: disk_wipe_mode
      object: baremetalhost
      annotation: "bmh.metal3.io/disk_wipe_mode"
    objectNames:
    - key: name
      object: machine
    - key: local-hostname
      object: machine
    - key: local_hostname
      object: machine
    - key: metal3-name
      object: baremetalhost
    - key: metal3-namespace
      object: baremetalhost
  networkData:
    links:
      ethernets:
      - id: enp1s0
        macAddress:
          fromHostInterface: enp1s0
        type: phy
    networks:
      ipv4:
      - id: baremetalv4
        ipAddressFromIPPool: my-ip-pool
        link: enp1s0
        routes:
        - gateway:
            fromIPPool: my-ip-pool
          network: 0.0.0.0
          prefix: 0
    services:
      dns:
      - 8.8.8.8

Note: When using spec.image.url with the oci:// prefix, you don’t need to pass the oci_image annotation through Metal3DataTemplate. The hardware manager reads directly from instance_info.image_source. This is the recommended approach for newer deployments.

Building an IPA Image with the Hardware Manager

To use this hardware manager, you need to build a custom IPA ramdisk image using ironic-python-agent-builder. This tool uses diskimage-builder (DIB) to create bootable ramdisk images containing the IPA and any custom elements you need.

Required Packages

The hardware manager requires several packages to be present in the IPA ramdisk:

Custom DIB Elements

DIB elements are modular components that customize the image build. Each element is a directory containing scripts that run at different phases of the build:

Scripts are named with a numeric prefix (e.g., 50-crane) to control execution order.

crane/
├── element-deps
└── install.d/
    └── 50-crane

#!/bin/bash

# https://docs.openstack.org/diskimage-builder/latest/developer/developing_elements.html

if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
    set -x
fi

set -eu
set -o pipefail

CRANE_VERSION="${DIB_CRANE_VERSION:-latest}"

# Detect architecture
ARCH=$(uname -m)
case "${ARCH}" in
    x86_64)
        CRANE_ARCH="x86_64"
        ;;
    aarch64)
        CRANE_ARCH="arm64"
        ;;
    *)
        echo "Unsupported architecture: ${ARCH}"
        exit 1
        ;;
esac

echo "Installing crane (${CRANE_VERSION}) for ${CRANE_ARCH}..."

# Get the download URL
if [ "${CRANE_VERSION}" = "latest" ]; then
    DOWNLOAD_URL=$(curl -s https://api.github.com/repos/google/go-containerregistry/releases/latest |
        grep "browser_download_url.*Linux_${CRANE_ARCH}.tar.gz" |
        cut -d '"' -f 4)
else
    DOWNLOAD_URL="https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${CRANE_ARCH}.tar.gz"
fi

if [ -z "${DOWNLOAD_URL}" ]; then
    echo "Failed to determine crane download URL"
    exit 1
fi

echo "Downloading crane from: ${DOWNLOAD_URL}"

# Download and extract crane
TEMP_DIR=$(mktemp -d)
curl -sL "${DOWNLOAD_URL}" | tar -xz -C "${TEMP_DIR}"

# Install crane binary
install -m 755 "${TEMP_DIR}/crane" /usr/local/bin/crane

# Cleanup
rm -rf "${TEMP_DIR}"

# Verify installation
if crane version; then
    echo "crane installed successfully"
else
    echo "crane installation verification failed"
    exit 1
fi

packages-install/
├── element-deps
└── install.d/
    └── 50-packages-install

#!/bin/bash

# https://docs.openstack.org/diskimage-builder/latest/developer/developing_elements.html

if [ "${DIB_DEBUG_TRACE:-0}" -gt 0 ]; then
    set -x
fi

set -eu
set -o pipefail

# Enable CRB (CodeReady Builder) repository and install EPEL
echo "Enabling CRB repository..."
dnf config-manager --set-enabled crb || true

# Detect CentOS version and install appropriate EPEL
if [ -f /etc/os-release ]; then
    # shellcheck disable=SC1091
    . /etc/os-release
    case "${VERSION_ID%%.*}" in
        9)
            echo "Installing EPEL for CentOS 9..."
            dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm || true
            ;;
        10)
            echo "Installing EPEL for CentOS 10..."
            dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm || true
            ;;
        *)
            echo "Unknown CentOS version: ${VERSION_ID}, skipping EPEL installation"
            ;;
    esac
fi

if [ -z "${DIB_EXTRA_PACKAGES:-}" ]; then
    echo "No extra packages specified via DIB_EXTRA_PACKAGES, skipping"
    exit 0
fi

echo "Updating system packages..."
dnf update -y

echo "Installing extra packages: ${DIB_EXTRA_PACKAGES}"

# shellcheck disable=SC2086
dnf install -y ${DIB_EXTRA_PACKAGES}

echo "Cleaning package cache..."
dnf clean all

echo "Extra packages installation complete"

Building the Image

Set the ELEMENTS_PATH to include your custom elements directory, then run the builder:

export ELEMENTS_PATH="/path/to/your/dib-elements"

export DIB_EXTRA_PACKAGES="jq yq mdadm lvm2 curl parted util-linux \
    squashfs-tools xfsprogs dosfstools grub2-efi-x64 grub2-tools rsync"

ironic-python-agent-builder \
    -o ipa-custom \
    -e extra-hardware \
    -e crane \
    -e packages-install \
    --release 9-stream centos

This produces two files:

• ipa-custom.kernel - The Linux kernel
• ipa-custom.initramfs - The ramdisk containing IPA and tools

For ARM64 builds, the grub packages differ:

export DIB_EXTRA_PACKAGES="jq yq mdadm lvm2 curl parted util-linux \
    squashfs-tools xfsprogs dosfstools grub2-efi-aa64 grub2-tools rsync"

Installing the Hardware Manager

The hardware manager must be placed in the IPA hardware managers directory and registered in setup.cfg.

File location:

ironic_python_agent/hardware_managers/deb_oci_efi_lvm.py

setup.cfg entry point:

Add the following entry to the ironic_python_agent.hardware_managers section in setup.cfg:

[entry_points]
ironic_python_agent.hardware_managers =
    deb_oci_efi_lvm = ironic_python_agent.hardware_managers.deb_oci_efi_lvm:DebOCIEFILVMHardwareManager

This registers the hardware manager as a plugin, allowing IPA to discover and load it at runtime.

Source Code

The implementation is shown below in expandable sections. Full source: deb_oci_efi_lvm.py.

Note: The code below uses a custom run_command helper function instead of IPA’s built-in ironic_python_agent.utils.execute. This was a deliberate choice to minimize dependencies on IPA internals, avoiding the need to keep the hardware manager in constant sync with IPA changes. However, reusing IPA’s existing utilities is a valid alternative approach.