Introducing the Metal3 IP Address Manager
As a part of developing the Cluster API Provider Metal3 (CAPM3) v1alpha4 release, the Metal3 crew introduced a new project: its own IP Address Manager. This blog post will go through the motivations behind such a project, the features that it brings, its use in Metal3 and future work.
What is the IP Address Manager?
The IP Address Manager (IPAM) is a controller that provides IP addresses and manages the allocations of IP subnets. It is not a DHCP server in that it only reconciles Kubernetes objects and does not answer any DHCP queries. It allocates IP addresses on request but does not handle any use of those addresses.
This sounds like the description of any IPAM system, no? Well, the twist is that this manager is based on Kubernetes to specifically handle some constraints from Metal3. We will go through the different issues that this project tackles.
When deploying nodes in a bare metal environment, there are a lot of possible variations. This project specifically aims to solve cases where static IP address configurations are needed. It is designed to specifically address this in the Cluster API (CAPI) context.
CAPI addresses the deployment of Kubernetes clusters and nodes, using the Kubernetes API. As such, it uses objects such as Machine Deployments (similar to deployments for pods) that takes care of creating the requested number of machines, based on templates. The replicas can be increased by the user, triggering the creation of new machines based on the provided templates. This mechanism does not allow for flexibility to be able to provide static addresses for each machine. The manager adds this flexibility by providing the address right before provisioning the node.
In addition, all the resources from the source cluster must support the CAPI pivoting, i.e. being copied and recreated in the target cluster. This means that all objects must contain all needed information in their spec field to recreate the status in the target cluster without losing information. All objects must, through a tree of owner references, be attached to the cluster object, for the pivoting to proceed properly.
In a nutshell, the manager provides an IP Address allocation service, based on Kubernetes API and fulfilling the needs of Metal3, specifically the requirements of CAPI.
How does it work?
The manager follows the same logic as the volume allocation in Kubernetes,
with a claim and an object created for that claim. There are three types of
objects defined, the
IPClaim and the
IPPool objects contain the definition of the IP subnets from which the
Addresses are allocated. It supports both IPv4 and IPv6. The subnets can either
be defined as such or given as start and end IP addresses with a prefix.
It also supports pre-allocating IP addresses.
The following is an example
IPPool definition :
apiVersion: ipam.metal3.io/v1alpha1 kind: IPPool metadata: name: pool1 spec: clusterName: cluster1 pools: - start: 192.168.0.10 end: 192.168.0.30 prefix: 25 gateway: 192.168.0.1 - subnet: 192.168.1.1/26 - subnet: 192.168.1.128/25 prefix: 24 gateway: 192.168.1.1 preAllocations: claim2: 192.168.0.12
IPPool would be defined similarly :
apiVersion: ipam.metal3.io/v1alpha1 kind: IPPool metadata: name: pool1 spec: clusterName: cluster1 pools: - start: 2001:0db8:85a3:0000:0000:8a2e::10 end: 2001:0db8:85a3:0000:0000:8a2e:ffff:fff0 prefix: 96 gateway: 12001:0db8:85a3:0000:0000:8a2e::1 - subnet: 2001:0db8:85a3:0000:0000:8a2d::/96 prefix: 96 gateway: 2001:0db8:85a3:0000:0000:8a2d::1
Whenever something requires an IP address from the
IPPool, it will create an
IPClaim contains a pointer to the
IPPool and an owner reference
to the object that created it.
The following is an example of an
apiVersion: ipam.metal3.io/v1alpha1 kind: IPClaim metadata: name: claim1 spec: pool: Name: pool1 status: address: Name: pool1-192-168-0-13
The controller will then reconcile this object and allocate an IP address. It
will create an
IPAddress object representing the allocated address. It will
then update the
IPPool status to list the IP Address and the
to point to the
The following is an example of an
apiVersion: ipam.metal3.io/v1alpha1 kind: IPAddress metadata: name: pool1-192-168-0-13 spec: pool: Name: pool1 claim: Name: claim1 address: 192.168.0.13 prefix: 24 gateway: 192.168.0.1
After this allocation, the
IPPool will be looking like this:
apiVersion: ipam.metal3.io/v1alpha1 kind: IPPool metadata: name: pool1 spec: clusterName: cluster1 pools: - start: 192.168.0.10 end: 192.168.0.30 prefix: 25 gateway: 192.168.0.1 - subnet: 192.168.1.1/26 - subnet: 192.168.1.128/25 prefix: 24 gateway: 192.168.1.1 preAllocations: claim2: 192.168.0.12 status: indexes: claim1: 192.168.0.13 claim2: 192.168.0.12
Use in Metal3
The IP Address Manager is used in Metal3 together with the metadata and network
data templates feature. Each Metal3Machine (M3M) and Metal3MachineTemplate
(M3MT) is associated with a Metal3DataTemplate that contains metadata and /
or a network data template that will be rendered for each Metal3Machine. The
rendered data will then be provided to Ironic. Those templates reference
IPPool objects. For each Metal3Machine, an
IPClaim is created for each
IPPool, and the templates are rendered with the allocated
This is how we achieve dynamic IP Address allocations in setups that require static configuration, allowing us to use Machine Deployment and Kubeadm Control Plane objects from CAPI in hardware labs where DHCP is not supported.
IPAddress has an owner reference set to its
IPClaim object, and
IPClaim objects have an owner reference set to the Metal3Data object created
from the Metal3DataTemplate, the owner reference chain links a Metal3Machine to
IPAddress objects were created for it, allowing for CAPI
The project is fulfilling its basic requirements, but we are looking into extending it and covering more use cases. For example, we are looking at adding integration with Infoblox and other external IPAM services. Do not hesitate to open an issue if you have some ideas for new features!
The project can be found here.