diff --git a/examples/terraform/bootkube-install/README.md b/examples/terraform/bootkube-install/README.md index a23c154c..266f5b4c 100644 --- a/examples/terraform/bootkube-install/README.md +++ b/examples/terraform/bootkube-install/README.md @@ -1,6 +1,6 @@ # Kubernetes -The Kubernetes example shows how to use Matchbox to network boot and provision a 3 node Kubernetes v1.8.5 cluster. This example uses [Terraform](https://www.terraform.io/intro/index.html) and a module provided by [Typhoon](https://github.com/poseidon/typhoon) to describe cluster resources. [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube) is run once to bootstrap the Kubernetes control plane. +The Kubernetes example shows how to use Matchbox to network boot and provision a 3 node Kubernetes v1.9.3 cluster. This example uses [Terraform](https://www.terraform.io/intro/index.html) and a module provided by [Typhoon](https://github.com/poseidon/typhoon) to describe cluster resources. [kubernetes-incubator/bootkube](https://github.com/kubernetes-incubator/bootkube) is run once to bootstrap the Kubernetes control plane. ## Requirements @@ -73,37 +73,57 @@ You may set certain optional variables to override defaults. Set `networking` to The default is to create a Kubernetes cluster with 1 controller and 2 workers as an example, but check `multi-controller.tfvars.example` for an example which defines 3 controllers and 1 worker. -## Apply +## Terraform -Fetch the `source` Terraform [module](https://www.terraform.io/docs/modules/index.html). +Initialize Terraform from the `bootkube-install` directory. ```sh -$ terraform get --update +terraform init ``` -Plan and apply to create the resources on Matchbox. +Get or update Terraform modules. + +```sh +$ terraform get # downloads missing modules +$ terraform get --update # updates all modules +Get: git::https://github.com/poseidon/typhoon (update) +Get: git::https://github.com/poseidon/bootkube-terraform.git?ref=v0.10.0 (update) +``` + +Plan the resources to be created. ```sh $ terraform plan -Plan: 37 to add, 0 to change, 0 to destroy. +Plan: 55 to add, 0 to change, 0 to destroy. ``` Terraform will configure matchbox with profiles (e.g. `cached-container-linux-install`, `bootkube-controller`, `bootkube-worker`) and add groups to match machines by MAC address to a profile. These resources declare that each machine should PXE boot and install Container Linux to disk. `node1` will provision itself as a controller, while `node2` and `node3` provision themselves as workers. The module referenced in `cluster.tf` will also generate bootkube assets to `assets_dir` (exactly like the [bootkube](https://github.com/kubernetes-incubator/bootkube) binary would). These assets include Kubernetes bootstrapping and control plane manifests as well as a kubeconfig you can use to access the cluster. +### ssh-agent + +Initial bootstrapping requires `bootkube.service` be started on one controller node. Terraform uses `ssh-agent` to automate this step. Add your SSH private key to `ssh-agent`, otherwise `terraform apply` will hang. + +```sh +ssh-add ~/.ssh/id_rsa +ssh-add -L +``` + +### Apply + +Apply the changes. + ```sh -$ terraform apply module.cluster.null_resource.copy-secrets.0: Still creating... (5m0s elapsed) module.cluster.null_resource.copy-secrets.1: Still creating... (5m0s elapsed) module.cluster.null_resource.copy-secrets.2: Still creating... (5m0s elapsed) ... module.cluster.null_resource.bootkube-start: Still creating... (8m40s elapsed) ... -Apply complete! Resources: 37 added, 0 changed, 0 destroyed. ``` -You can now move on to the "Machines" section. Apply will loop until it can successfully copy the kubeconfig and etcd TLS assets to each node and start the one-time Kubernetes bootstrapping process on a controller. In practice, you may see `apply` fail if it connects before the disk install has completed. Run terraform apply until it reconciles successfully. +Apply will then loop until it can successfully copy credentials to each machine and start the one-time Kubernetes bootstrap service. Proceed to the next step while this loops. ## Machines @@ -129,26 +149,26 @@ $ sudo ./scripts/libvirt [start|reboot|shutdown|poweroff|destroy] $ export KUBECONFIG=assets/auth/kubeconfig $ kubectl get nodes NAME STATUS AGE VERSION -node1.example.com Ready 11m v1.8.5 -node2.example.com Ready 11m v1.8.5 -node3.example.com Ready 11m v1.8.5 +node1.example.com Ready 11m v1.9.3 +node2.example.com Ready 11m v1.9.3 +node3.example.com Ready 11m v1.9.3 $ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE -kube-system checkpoint-installer-p8g8r 1/1 Running 1 13m -kube-system kube-apiserver-s5gnx 1/1 Running 1 41s -kube-system kube-controller-manager-3438979800-jrlnd 1/1 Running 1 13m -kube-system kube-controller-manager-3438979800-tkjx7 1/1 Running 1 13m -kube-system kube-dns-4101612645-xt55f 4/4 Running 4 13m -kube-system kube-flannel-pl5c2 2/2 Running 0 13m -kube-system kube-flannel-r9t5r 2/2 Running 3 13m -kube-system kube-flannel-vfb0s 2/2 Running 4 13m -kube-system kube-proxy-cvhmj 1/1 Running 0 13m -kube-system kube-proxy-hf9mh 1/1 Running 1 13m -kube-system kube-proxy-kpl73 1/1 Running 1 13m -kube-system kube-scheduler-694795526-1l23b 1/1 Running 1 13m -kube-system kube-scheduler-694795526-fks0b 1/1 Running 1 13m -kube-system pod-checkpointer-node1.example.com 1/1 Running 2 10m +kube-system kube-flannel-fqp7f 2/2 Running 1 11m +kube-system kube-flannel-gnjrm 2/2 Running 0 11m +kube-system kube-flannel-llbgt 2/2 Running 0 11m +kube-system kube-apiserver-7336w 1/1 Running 0 11m +kube-system kube-controller-manager-3271970485-b9chx 1/1 Running 0 11m +kube-system kube-controller-manager-3271970485-v30js 1/1 Running 1 11m +kube-system kube-dns-1187388186-mx9rt 3/3 Running 0 11m +kube-system kube-proxy-50sd4 1/1 Running 0 11m +kube-system kube-proxy-bczhp 1/1 Running 0 11m +kube-system kube-proxy-mp2fw 1/1 Running 0 11m +kube-system kube-scheduler-3895335239-fd3l7 1/1 Running 1 11m +kube-system kube-scheduler-3895335239-hfjv0 1/1 Running 0 11m +kube-system pod-checkpointer-wf65d 1/1 Running 0 11m +kube-system pod-checkpointer-wf65d-node1.example.com 1/1 Running 0 11m ``` ## Addons diff --git a/examples/terraform/bootkube-install/cluster.tf b/examples/terraform/bootkube-install/cluster.tf index 274467e8..99e67045 100644 --- a/examples/terraform/bootkube-install/cluster.tf +++ b/examples/terraform/bootkube-install/cluster.tf @@ -1,6 +1,6 @@ // Kubernetes cluster module "cluster" { - source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.8.5" + source = "git::https://github.com/poseidon/typhoon//bare-metal/container-linux/kubernetes?ref=v1.9.3" providers = { local = "local.default"