Manage a fleet with OpenShift Advanced Cluster Management
Herein I will describe how Open Cluster Management (OCM) works and how OpenShift’s MultiCluster Engine (MCE) and Advanced Cluster Management (ACM) build on it to manage a fleet of clusters and distribute work to them.
When managing a fleet of Kubernetes clusters it eventually becomes desirable to manage them as a unit and apply configuration and packages once to all and/or subsets of clusters. Activities and configurations often managed at fleet level include the following:
- Work scheduling: deploy and monitor workloads and resources on any cluster
- Policy enforcement: apply and monitor platform and runtime policies for any cluster
- Identity and access management: use consistent authentication and authorization rules in all clusters
- Command and control: inventory and observe resources in all clusters
- Cluster lifecycle: provision new clusters and update existing ones
Commonly, a hub-and-spoke topology is implemented to provide this fleet management. Work, policy, identity and other configuration are defined in the hub cluster and somehow automatically distributed to selected spokes.
OCM
Open Cluster Management (OCM) is a framework for creating such a hub-and-spoke topology and distributing work. The hub cluster runs cluster-manager to register and monitor spoke clusters. Each spoke cluster runs an agent named klusterlet which registers with cluster-manager and pulls ManifestWork resources from a namespace dedicated to it in the hub.
Note: A hub-and-spoke topology can be implemented with controllers in the hub actively managing spoke clusters, also known as a “push” model; or with controllers in the spoke clusters pulling and applying configuration from the hub, also known as a “pull” model. The “pull” model generally scales better by distributing some load to each spoke, and OCM emphasizes the pull model for scalability.
The flow at the core of OCM is: Kubernetes resources are wrapped in a ManifestWork resource and the ManifestWork is placed in a namespace dedicated to a spoke cluster. The klusterlet in the spoke cluster calls the hub’s API server to retrieve this ManifestWork and applies it to itself as AppliedManifestWork.
ManifestWork is an implementation of SIG-Multicluster’s Work API. In a way it’s like a basic “GitOps” workflow - declare the resource and have it reconciled in the spoke cluster automatically.
One way of using OCM is to craft ManifestWorks and place them in selected namespaces as described here. ManifestWorkReplicaSet is a feature in development to make such distribution easier.
Some commands for inspection:
- See all ManifestWorks published for a spoke cluster:
clusteradm get works --cluster <cluster-name>oroc get manifestworks.work.open-cluster-management.io -n <cluster-name>. - Find works that have been applied in a spoke cluster with
oc get appliedmanifestworks. - View logs showing the klusterlet pulling its work with
oc logs -n open-cluster-management-agent deployments/klusterlet-agent.
MultiCluster Engine (MCE) and Advanced Cluster Management (ACM)
In OpenShift, OCM is installed by the MultiClusterEngine (MCE) operator and custom resource as implemented in stolostron/backplane-operator. MCE installs OCM, several OCM AddOns used in cluster-lifecycle, and cluster-lifecycle operators such as Hive, cluster-api-installer, and HyperShift - find the full list here.
MultiClusterEngine is often installed as a component of Advanced Cluster Management, which provides many more OCM addons. ACM is installed by the MultiClusterHub operator and custom resource as implemented in stolostron/multiclusterhub-operator. It includes all of MCE plus AddOns for workload scheduling with ArgoCD, policy application with Gatekeeper, multicluster observability with Grafana, cross-cluster resource search, cross-cluster networking with Submariner, and hub disaster recovery with Velero/OADP. The full list is here.
The simplest way to manage the components installed by MCE and ACM is to edit the MultiClusterEngine and MultiClusterHub resources respectively.
OCM Addons
While it’s possible to work with ManifestWork directly, AddOns are a key feature of OCM that abstract ManifestWorks into higher-level CRDs managed by special controllers. AddOns are installed via independent Helm charts or using the MCE and MCH CRs mentioned above. To quickly see available and enabled AddOns run clusteradm get addon --output table.
AddOn managers are responsible for managing several component types. They deploy hub-side components like proxy ingress servers and GitOps controllers. They deploy spoke-side components like proxy clients and search indexers. And they watch for custom CRDs to dynamically reconcile into ManifestWorks to be put to spoke clusters.
The Governance Policy Framework is an example of several AddOns that together publish controllers in both hub and spoke clusters and handle custom CRDs like CertificatePolicy, ConfigurationPolicy and OperatorPolicy.
Cluster Proxy is another AddOn. It deploys a ManifestWork with a Deployment, Secrets and other resources to configure a Konnectivity client in the managed cluster. This communicates with a hub-side proxy server which it also configures. You can find the deployed resources yourself in a simple hub cluster with oc get manifestwork -n local-cluster addon-cluster-proxy-deploy-0. cluster-proxy also watches the high-level CRD ManagedProxyConfiguration and updates its deployments.
The following AddOns are available in a basic ACM cluster:
| Name | Role |
|---|---|
| application-manager | deploy applications to spoke clusters using channels and subscriptions (deprecated) |
| cert-policy-controller | apply CertificatePolicy resources to spoke clusters |
| cluster-proxy | enable hub cluster to communicate with spoke clusters via proxy |
| config-policy-controller | apply ConfigurationPolicy resources to spoke clusters |
| gitops-addon | apply ArgoCD ApplicationSets to spoke clusters via pull model |
| governance-policy-framework | apply typed manifests to managed clusters |
| governance-standalone-hub-templating | ? |
| hypershift-addon | integrate OCM with hosted control plane clusters |
| managed-serviceaccount | share a serviceaccount from managed clusters to hub |
| search-collector | gather resources from managed cluster for fast search from hub |
| submariner | connect managed clusters into a flat network |
| volsync | backup persistent volumes between clusters |
| work-manager | provides high-level structures for execution on managed clusters |
Conclusion
OCM is a valuable framework for distributing work across a fleet. It is pull-oriented for scalability, standards-compliant for compatibility, and extensible for flexibility and ease of use.