diff --git a/docs/content/cluster-api/cluster-class.md b/docs/content/cluster-api/cluster-class.md index da8b116..10e7647 100644 --- a/docs/content/cluster-api/cluster-class.md +++ b/docs/content/cluster-api/cluster-class.md @@ -1,104 +1,642 @@ -# Cluster Class +# Cluster Class with Kamaji -Kamaji supports **ClusterClass**, a simple way to create many clusters of a similar shape. This is useful for creating many clusters with the same configuration, such as a development cluster, a staging cluster, and a production cluster. +`ClusterClass` is a Cluster API feature that enables template-based cluster creation. When combined with Kamaji's hosted control plane architecture, `ClusterClass` provides a powerful pattern for standardizing Kubernetes cluster deployments across multiple infrastructure providers while maintaining consistent control plane configurations. !!! warning "Experimental Feature" - ClusterClass is an experimental feature of Cluster API. As with any experimental features it should be used with caution as it may be unreliable. All experimental features are not subject to any compatibility or deprecation policy and are not yet recommended for production use. + ClusterClass is still an experimental feature of Cluster API. As with any experimental features it should be used with caution. Read more about ClusterClass in the [Cluster API documentation](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-class/). -You can read more about ClusterClass in the [Cluster API documentation](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-class/). +## Understanding Cluster Class -## Enabling ClusterClass +`ClusterClass` reduces configuration boilerplate by defining reusable cluster templates. Instead of creating individual resources for each cluster, you define a `ClusterClass` once and create multiple clusters from it with minimal configuration. -To enable ClusterClass, you need to set `CLUSTER_TOPOLOGY` before running `clusterctl init`. This will enable the Cluster API feature gate for ClusterClass. +With Kamaji, this pattern becomes even more powerful: +- **Shared Control Plane Templates**: The same KamajiControlPlaneTemplate works across all infrastructure providers +- **Infrastructure Flexibility**: Deploy worker nodes on vSphere, AWS, Azure, or any supported provider while maintaining consistent control planes +- **Simplified Management**: Hosted control planes reduce the complexity of `ClusterClass` templates + +## Enabling Cluster Class + +To use `ClusterClass` with Kamaji, you need to enable the cluster topology feature gate before initializing the management cluster: ```bash export CLUSTER_TOPOLOGY=true -clusterctl init --infrastructure vsphere --control-plane kamaji +clusterctl init --control-plane kamaji --infrastructure vsphere ``` -## Creating a ClusterClass +This will install: +- Cluster API core components with `ClusterClass` support +- Kamaji Control Plane Provider +- Your chosen infrastructure provider (vSphere in this example) -To create a ClusterClass, you need to create a `ClusterClass` custom resource. Here is an example of a `ClusterClass` that will create a cluster running control plane on the Kamaji Management Cluster and worker nodes on vSphere: +Verify the installation: + +```bash +kubectl get deployments -A | grep -E "capi|kamaji" +``` + +## Template Architecture with Kamaji + +A `ClusterClass` with Kamaji consists of four main components: + +1. Control Plane Template (KamajiControlPlaneTemplate): Defines the hosted control plane configuration that remains consistent across infrastructure providers. + +2. Infrastructure Template (VSphereClusterTemplate): Provider-specific infrastructure configuration for the cluster. + +3. Bootstrap Template (KubeadmConfigTemplate): Node initialization configuration that works across providers. + +4. Machine Template (VSphereMachineTemplate): Provider-specific machine configuration for worker nodes. + +Here's how these components relate in a `ClusterClass`: ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: ClusterClass metadata: - name: kamaji-clusterclass + name: kamaji-vsphere-class spec: - controlPlane: - ref: - apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 - kind: KamajiControlPlaneTemplate - name: kamaji-clusterclass-kamaji-control-plane-template + # Infrastructure provider template infrastructure: ref: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereClusterTemplate - name: kamaji-clusterclass-vsphere-cluster-template + name: vsphere-cluster-template + + # Kamaji control plane template - reusable across providers + controlPlane: + ref: + apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 + kind: KamajiControlPlaneTemplate + name: kamaji-control-plane-template + + # Worker configuration workers: machineDeployments: - - class: kamaji-clusterclass + - class: default-worker template: bootstrap: ref: apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 kind: KubeadmConfigTemplate - name: kamaji-clusterclass-kubeadm-config-template + name: worker-bootstrap-template infrastructure: ref: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: VSphereMachineTemplate - name: kamaji-clusterclass-vsphere-machine-template - -# other resources omitted for brevity ... + name: vsphere-worker-template ``` -The template file [`capi-kamaji-vsphere-class-template.yaml`](https://raw.githubusercontent.com/clastix/cluster-api-control-plane-provider-kamaji/master/templates/vsphere/capi-kamaji-vsphere-class-template.yaml) provides a full example of a ClusterClass for vSphere. You can generate a ClusterClass manifest using `clusterctl`. +The key advantage: the KamajiControlPlaneTemplate and KubeadmConfigTemplate can be shared across different infrastructure providers, while only the infrastructure-specific templates need to change. -Before you need to list all the variables in the template file: +## Creating a Cluster Class -```bash -cat capi-kamaji-vsphere-class-template.yaml | clusterctl generate yaml --list-variables +Let's create a `ClusterClass` for vSphere with Kamaji. First, define the shared templates: + +### KamajiControlPlaneTemplate + +This template defines the hosted control plane configuration: + +```yaml +apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 +kind: KamajiControlPlaneTemplate +metadata: + name: kamaji-controlplane + namespace: capi-templates-vsphere +spec: + template: + spec: + dataStoreName: "default" # Default datastore for etcd + + network: + serviceType: LoadBalancer + serviceAddress: "" + certSANs: [] + + addons: + coreDNS: {} + kubeProxy: {} + konnectivity: {} + + apiServer: + extraArgs: [] + resources: + requests: {} + controllerManager: + extraArgs: [] + resources: + requests: {} + scheduler: + extraArgs: [] + resources: + requests: {} + + kubelet: + cgroupfs: systemd + preferredAddressTypes: + - InternalIP + + registry: "registry.k8s.io" ``` -Fill them with the desired values and generate the manifest: +### KubeadmConfigTemplate -```bash -clusterctl generate yaml \ - --from capi-kamaji-vsphere-class-template.yaml \ - > capi-kamaji-vsphere-class.yaml +This bootstrap template configures worker nodes: + +```yaml +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: worker-bootstrap-template +spec: + template: + spec: + + # Configuration for kubeadm join + joinConfiguration: + discovery: {} + nodeRegistration: + criSocket: /var/run/containerd/containerd.sock + imagePullPolicy: IfNotPresent + name: '{{ local_hostname }}' + kubeletExtraArgs: + cloud-provider: external + node-ip: "{{ ds.meta_data.local_ipv4 }}" + + # Commands to run before kubeadm join + preKubeadmCommands: + - hostnamectl set-hostname "{{ ds.meta_data.hostname }}" + - echo "127.0.0.1 {{ ds.meta_data.hostname }}" >> /etc/hosts + + # Commands to run after kubeadm join + postKubeadmCommands: [] + + # Users to create on worker nodes + users: [] ``` -Apply the generated manifest to create the ClusterClass: +### VSphereClusterTemplate -```bash -kubectl apply -f capi-kamaji-vsphere-class.yaml +Infrastructure-specific template for vSphere: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: VSphereClusterTemplate +metadata: + name: vsphere + namespace: capi-templates-vsphere +spec: + template: + spec: + server: "vcenter.sample.com" # vCenter server address + thumbprint: "" # vCenter certificate thumbprint + + identityRef: + kind: VSphereClusterIdentity + name: "vsphere-cluster-identity" + + failureDomainSelector: {} + clusterModules: [] ``` -## Creating a Cluster from a ClusterClass +### VSphereMachineTemplate -Once a ClusterClass is created, you can create a Cluster using the ClusterClass. Here is an example of a Cluster that uses the `kamaji-clusterclass`: +Machine template for vSphere workers: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: VSphereMachineTemplate +metadata: + name: vsphere-vm-base + namespace: capi-templates-vsphere +spec: + template: + spec: + # Resources will be patched by ClusterClass based on variables + # numCPUs, memoryMiB, diskGiB are dynamically set + + # Infrastructure defaults - will be patched by ClusterClass + server: "vcenter.sample.com" + datacenter: "datacenter" + datastore: "datastore" + resourcePool: "Resources" + folder: "vm-folder" + template: "ubuntu-2404-kube-v1.32.0" + storagePolicyName: "" + thumbprint: "" + + # Network configuration (IPAM by default) + network: + devices: + - networkName: "k8s-network" + dhcp4: false + addressesFromPools: + - apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: "{{ .builtin.cluster.name }}" # Uses cluster name +``` + +### Variables and Patching in Cluster Class + +`ClusterClass` becomes powerful through its variable system and JSON patching capabilities. This allows the same templates to be customized for different use cases without duplicating YAML. + +#### Variable System + +Variables in `ClusterClass` define the parameters users can customize when creating clusters. Each variable has: + +- **Schema Definition**: OpenAPI v3 schema that validates input +- **Required/Optional**: Whether the variable must be provided +- **Default Values**: Fallback values when not specified +- **Type Constraints**: Data types, ranges, and enum values + +Here's how variables work in practice: + +**Control Plane Variables:** +```yaml +variables: +- name: kamajiControlPlane + required: true + schema: + openAPIV3Schema: + type: object + properties: + dataStoreName: + type: string + description: "Datastore name for etcd" + default: "default" + network: + type: object + properties: + serviceType: + type: string + enum: ["ClusterIP", "NodePort", "LoadBalancer"] + default: "LoadBalancer" + serviceAddress: + type: string + description: "Pre-assigned VIP address" +``` + +**Machine Resource Variables:** +```yaml +- name: machineSpecs + required: true + schema: + openAPIV3Schema: + type: object + properties: + numCPUs: + type: integer + minimum: 2 + maximum: 64 + default: 4 + memoryMiB: + type: integer + minimum: 4096 + maximum: 131072 + default: 8192 + diskGiB: + type: integer + minimum: 40 + maximum: 2048 + default: 100 +``` + +#### JSON Patching System + +Patches apply variable values to the base templates at cluster creation time. This enables the same template to serve different configurations. + +**Control Plane Patching:** +```yaml +patches: +- name: controlPlaneConfig + definitions: + - selector: + apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 + kind: KamajiControlPlaneTemplate + matchResources: + controlPlane: true + jsonPatches: + - op: replace + path: /spec/template/spec/dataStoreName + valueFrom: + variable: kamajiControlPlane.dataStoreName + - op: replace + path: /spec/template/spec/network/serviceType + valueFrom: + variable: kamajiControlPlane.network.serviceType +``` + +**Machine Resource Patching:** +```yaml +- name: machineResources + definitions: + - selector: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: VSphereMachineTemplate + matchResources: + machineDeploymentClass: + names: ["default-worker"] + jsonPatches: + - op: add # Resources are not in base template + path: /spec/template/spec/numCPUs + valueFrom: + variable: machineSpecs.numCPUs + - op: add + path: /spec/template/spec/memoryMiB + valueFrom: + variable: machineSpecs.memoryMiB +``` + +#### Advanced Patching Patterns + +**Conditional Patching:** +```yaml +- name: optionalVIP + definitions: + - selector: + apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 + kind: KamajiControlPlaneTemplate + jsonPatches: + - op: replace + path: /spec/template/spec/network/serviceAddress + valueFrom: + variable: kamajiControlPlane.network.serviceAddress + # Only applies if serviceAddress is not empty + enabledIf: "{{ ne .kamajiControlPlane.network.serviceAddress \"\" }}" +``` + +**Infrastructure Patching:** +```yaml +- name: infrastructureConfig + definitions: + - selector: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: VSphereMachineTemplate + jsonPatches: + - op: replace + path: /spec/template/spec/datacenter + valueFrom: + variable: infrastructure.datacenter + - op: replace + path: /spec/template/spec/datastore + valueFrom: + variable: infrastructure.datastore + - op: replace + path: /spec/template/spec/template + valueFrom: + variable: infrastructure.vmTemplate +``` + +### Complete Cluster Class with Variables + +For a comprehensive example with all variables and patches configured, see the [vsphere-kamaji-clusterclass.yaml](https://raw.githubusercontent.com/clastix/cluster-api-control-plane-provider-kamaji/master/templates/vsphere/capi-kamaji-vsphere-class-template.yaml) template. + +## Creating a Cluster from Cluster Class + +With the `ClusterClass` defined, creating a cluster becomes remarkably simple: ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: - name: sample + name: my-cluster + namespace: default spec: + # Network configuration defined at cluster level + clusterNetwork: + pods: + cidrBlocks: ["10.244.0.0/16"] + services: + cidrBlocks: ["10.96.0.0/12"] + serviceDomain: "cluster.local" + topology: - class: kamaji-clusterclass - classNamespace: capi-clusterclass - version: v1.31.0 + class: vsphere-standard + classNamespace: capi-templates-vsphere + version: v1.32.0 + controlPlane: replicas: 2 + workers: machineDeployments: - - class: kamaji-clusterclass - name: md-sample + - class: default-worker + name: worker-nodes replicas: 3 - -# other resources omitted for brevity ... + + variables: + - name: kamajiControlPlane + value: + dataStoreName: "etcd" + network: + serviceType: "LoadBalancer" + serviceAddress: "" # Auto-assigned if empty + + - name: machineSpecs + value: + numCPUs: 8 + memoryMiB: 16384 + diskGiB: 60 + + - name: infrastructure + value: + vmTemplate: "ubuntu-2404-kube-v1.32.0" + datacenter: "K8s-TI-dtc" + datastore: "K8s-N01td-01" + resourcePool: "rp-kamaji-dev" + folder: "my-cluster-vms" + + - name: networking + value: + networkName: "VM-K8s-TI-cpmgmt" + nameservers: ["8.8.8.8", "1.1.1.1"] + dhcp4: false # Using IPAM ``` -Always refer to the [Cluster API documentation](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-class/) for the most up-to-date information on ClusterClass. \ No newline at end of file +Create the cluster: + +```bash +kubectl apply -f my-cluster.yaml +``` + +Monitor cluster creation: + +```bash +clusterctl describe cluster my-cluster +kubectl get cluster,kamajicontrolplane,machinedeployment -n default +``` + +With this approach, the same `KamajiControlPlaneTemplate` and `KubeadmConfigTemplate` can be reused when creating `ClusterClasses` for AWS, Azure, or any other provider. Only the infrastructure-specific templates need to change. + +## Cross-Provider Template Reuse + +One of Kamaji's key advantages with `ClusterClass` is template modularity across providers. Here's how to leverage this: + +### Shared Templates Repository + +Create a namespace for shared templates: + +```bash +kubectl create namespace cluster-templates +``` + +Deploy shared Kamaji and bootstrap templates once: + +```bash +kubectl apply -n cluster-templates -f kamaji-controlplane-template.yaml +kubectl apply -n cluster-templates -f kubeadm-config-template.yaml +``` + +### Provider-Specific Cluster Classes + +For each infrastructure provider, create a `ClusterClass` that references the shared templates: + +#### AWS Cluster Class + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: ClusterClass +metadata: + name: kamaji-aws-class +spec: + controlPlane: + ref: + apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 + kind: KamajiControlPlaneTemplate + name: kamaji-controlplane + namespace: cluster-templates # Shared template + + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta2 + kind: AWSClusterTemplate + name: aws-cluster-template # AWS-specific + + workers: + machineDeployments: + - class: default-worker + template: + bootstrap: + ref: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: kubeadm + namespace: cluster-templates # Shared template + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta2 + kind: AWSMachineTemplate + name: aws-worker-template # AWS-specific +``` + +### Azure Cluster Class + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: ClusterClass +metadata: + name: kamaji-azure-class +spec: + controlPlane: + ref: + apiVersion: controlplane.cluster.x-k8s.io/v1alpha1 + kind: KamajiControlPlaneTemplate + name: kamaji-control-plane-template + namespace: cluster-templates # Same shared template + + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureClusterTemplate + name: azure-cluster-template # Azure-specific + + workers: + machineDeployments: + - class: default-worker + template: + bootstrap: + ref: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: worker-bootstrap-template + namespace: cluster-templates # Same shared template + infrastructure: + ref: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: AzureMachineTemplate + name: azure-worker-template # Azure-specific +``` + +## Managing Cluster Class Lifecycle + +### Listing Available Cluster Classes + +```bash +kubectl get clusterclasses -A +``` + +### Viewing Cluster Class Details + +```bash +kubectl describe clusterclass vsphere-standard -n capi-templates-vsphere +``` + +### Updating a Cluster Class + +A `ClusterClass` update affects only new clusters. Existing clusters continue using their original configuration: + +```bash +kubectl edit clusterclass vsphere-standard -n capi-templates-vsphere +``` + +### Deleting Clusters Created from Cluster Class + +Always delete clusters before removing the `ClusterClass`: + +```bash +# Delete the cluster +kubectl delete cluster my-cluster + +# Wait for cleanup +kubectl wait --for=delete cluster/my-cluster --timeout=10m + +# Then safe to delete ClusterClass if no longer needed +kubectl delete clusterclass vsphere-standard -n capi-templates-vsphere +``` + +## Template Versioning Strategies + +When managing `ClusterClasses` across environments, consider these versioning approaches: + +### Semantic Versioning in Names + +```yaml +metadata: + name: vsphere-standard-v1-2-0 + namespace: capi-templates-vsphere +``` + +### Using Labels for Version Tracking + +```yaml +metadata: + name: vsphere-standard + namespace: capi-templates-vsphere + labels: + version: "1.2.0" + stability: "stable" + tier: "standard" +``` + +### Namespace Separation + +```bash +kubectl create namespace clusterclass-v1 +kubectl create namespace clusterclass-v2 +``` + +This enables gradual migration between `ClusterClass` versions while maintaining compatibility. + +## Further Reading + + - [Cluster API ClusterClass Documentation](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-class/) + - [Kamaji Control Plane Provider Reference](https://doc.crds.dev/github.com/clastix/cluster-api-control-plane-provider-kamaji) + - [CAPI Provider Integration](https://github.com/clastix/cluster-api-control-plane-provider-kamaji) \ No newline at end of file diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 528ea4e..c3c8bb1 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -63,11 +63,11 @@ nav: - 'Cluster API': - cluster-api/index.md - cluster-api/control-plane-provider.md - - cluster-api/cluster-class.md - - cluster-api/cluster-autoscaler.md - cluster-api/vsphere-infra-provider.md - cluster-api/proxmox-infra-provider.md - cluster-api/other-providers.md + - cluster-api/cluster-autoscaler.md + - cluster-api/cluster-class.md - 'Guides': - guides/index.md - guides/alternative-datastore.md