Blog

Read about the newest updates in the community.

Metal3 Introduces Pivoting

By Kashif Nizam Khan
metal3 baremetal pivoting move
Share on Twitter

Metal3 project has introduced pivoting in its CI workflow. The motivation for pivoting is to move all the objects from the ephemeral/management cluster to a target cluster. This blog post will briefly introduce the concept of pivoting and the impact it has on the overall CI workflow. For the rest of this blog, we refer ephemeral/management cluster as an ephemeral cluster.

What is Pivoting?

In the context of Metal3 Provider, Pivoting is the process of moving Cluster-API and Metal3 objects from the ephemeral k8s cluster to a target cluster. In Metal3, this process is performed using the clusterctl tool provided by Cluster-API. clusterctl recognizes pivoting as a move. During the pivot process, clusterctl pauses any reconciliation of Cluster-API objects and this gets propagated to Cluster-api-provider-metal3 (CAPM3) objects as well. Once all the objects are paused, the objects are created on the other side on the target cluster and deleted from the ephemeral cluster.

Prerequisites

Prior to the actual pivot process, the target cluster should already have the provider components, ironic containers and CNI installed and running. To perform pivot outside metal3-dev-env, specifically, the following points need to be addressed:

  • clusterctl is used to initialize both the ephemeral and target cluster.
  • BMH objects have correct status annotation.
  • Maintain connectivity towards the provisioning network.
  • Baremetal Operator(BMO) is deployed as part of CAPM3.
  • Objects should have a proper owner reference chain.

For a detailed explanation of the above-mentioned prerequisites please read the pivoting documentation.

Pivoting workflow in CI

The Metal3 CI currently includes pivoting as part of the deployment process both for Ubuntu and CentOS-based jobs. This essentially means all the PRs that go in, are tested through the pivoting workflow. Here is the CI deployment workflow:

  • make the metal3-dev-env. It gives us the ephemeral cluster with all the necessary controllers running within it. The corresponding metal3-dev-env command is make
  • provision target cluster. For normal integration tests, this step deploys a control-plane node and a worker in the target cluster. For, feature-test and feature-test-upgrade the provision step deploys three control-planes and a worker. The corresponding metal3-dev-env commands are (normal integration test workflow):
./scripts/provision/cluster.sh
./scripts/provision/controlplane.sh
./scripts/provision/worker.sh
  • Initialize the provider components on the target cluster. This installs all the controllers and associated components related to cluster-api , cluster-api-provider-metal3, baremetal-operator and ironic. Since it is necessary to have only one set of ironic deployment/containers in the picture, this step also deletes the ironic deployment/containers from ephemeral cluster.
  • Move all the objects from ephemeral to the target cluster.
  • Check the status of the objects to verify whether the objects are being reconciled correctly by the controllers in the target cluster. This step verifies and finalizes the pivoting process. The corresponding metal3-dev-env the command that performs this and the previous two steps is :
./scripts/feature_tests/pivoting/pivot.sh
  • Move the objects back to the ephemeral cluster. This step also removes the ironic deployment from the target cluster and reinstates the ironic deployment/containers in the ephemeral cluster. Since we do not delete the provider components in the ephemeral cluster, installing them again is not necessary. The corresponding metal3-dev-env command that performs this step is :
./scripts/feature_tests/pivoting/repivot.sh
  • De-provision the BMHs and delete the target cluster. The corresponding metal3-dev-env commands to de-provision worker, controlplane and the cluster is as follows:
./scripts/deprovision/worker.sh
./scripts/deprovision/controlplane.sh
./scripts/deprovision/cluster.sh

Note that, if we de-provision cluster, that would de-provision worker and controlplane automatically.

Pivoting in Metal3

The pivoting process described above is realized in ansible scripts move.yml and move_back.yml. Under the hood, pivoting uses the move command from clusterctl provided by Cluster-API.

As stated earlier, all the PRs that go into any Metal3 repository where the integration tests are run, the code change introduced in the PR is verified with pivoting also in the integration tests now. Moreover, the upgrade workflow in Metal3 performs all the upgrade operations in Metal3 after pivoting to the target cluster.