mirror of
https://github.com/outbackdingo/kamaji.git
synced 2026-01-27 10:19:29 +00:00
Update documentation with latest changes (#44)
* docs: set the kind guide as getting started * docs: update guides to latest changes * docs: minor add to README
This commit is contained in:
Adriano Pezzuto
17
README.md
17
README.md
@@ -43,19 +43,25 @@ A dedicated `etcd` cluster for each tenant cluster doesn’t scale well for a ma
|
||||
With this solution, the resiliency is guaranteed by the usual `etcd` mechanism, and the pods' count remains under control, so it solves the main goal of resiliency and costs optimization. The trade-off here is that we have to operate an external `etcd` cluster and manage the access to be sure that each tenant cluster uses only its data. Also, there are limits in size in `etcd`, defaulted to 2GB and configurable to a maximum of 8GB. We’re solving this issue by pooling multiple `etcd` and sharding the tenant control planes.
|
||||
|
||||
## Use cases
|
||||
Kamaji project has been initially started as a solution for actual and common problems such as minimizing the Total Cost of Ownership while running Kubernetes at scale. However, it can open a wider range of use cases. Here are a few:
|
||||
Kamaji project has been initially started as a solution for actual and common problems such as minimizing the Total Cost of Ownership while running Kubernetes at large scale. However, it can open a wider range of use cases. Here are a few:
|
||||
|
||||
### Managed Kubernetes
|
||||
Enabling companies to provide Cloud Native Infrastructure with ease by introducing a strong separation of concerns between management and workloads. Centralize clusters management, monitoring, and observability by leaving developers to focus on the applications, increase productivity and reduce operational costs.
|
||||
|
||||
### Kubernetes as a Service
|
||||
Provide Kubernetes clusters in a self-service fashion by running management and workloads on different infrastructures and cost centers with the option of Bring Your Own Device - BYOD.
|
||||
|
||||
### Control Plane as a Service
|
||||
Provide Kubernetes control plane in a self-service fashion by running management and workloads on different infrastructures and cost centers with the option of Bring Your Own Device - BYOD.
|
||||
Provide multiple Kubernetes control planes running on top of a single Kubernetes cluster. Tenants who use namespaces based isolation often still need access to cluster wide resources like Cluster Roles, Admission Webhooks, or Custom Resource Definitions.
|
||||
|
||||
### Edge Computing
|
||||
Distribute Kubernetes workloads across edge computing locations without having to manage multiple clusters across various providers. Centralize management of hundreds of control planes while leaving workloads to run isolated on their own dedicated infrastructure.
|
||||
|
||||
### Cluster Simulations
|
||||
Test a new Kubernetes API or experimental flag or a new tool without impacting production operations. Kamaji will let you simulate such things in a safe and controlled environment.
|
||||
### Cluster Simulation
|
||||
Check new Kubernetes API or experimental flag or a new tool without impacting production operations. Kamaji will let you simulate such things in a safe and controlled environment.
|
||||
|
||||
### Workloads Testing
|
||||
Check the behaviour of your workloads on different and multiple versions of Kubernetes with ease by deploying multiple Control Planes in a single cluster.
|
||||
|
||||
## Features
|
||||
|
||||
@@ -88,7 +94,6 @@ Tenant clusters are fully CNCF compliant built with upstream Kubernetes binaries
|
||||
- [ ] `kine` integration, i.e. use MySQL, SQLite, PostgreSQL as datastore
|
||||
- [ ] Deeper `kubeadm` integration
|
||||
- [ ] `etcd` pooling
|
||||
- [ ] Tenant Control Planes sharding
|
||||
|
||||
## Documentation
|
||||
Please, check the project's [documentation](./docs/) for getting started with Kamaji.
|
||||
@@ -122,4 +127,4 @@ A. Lighter Multi-Tenancy solutions, like Capsule shares the Kubernetes control p
|
||||
|
||||
Q. So I need a costly cloud infrastructure to try Kamaji?
|
||||
|
||||
A. No, it is possible to try Kamaji on your laptop with [KinD](./deploy/kind/README.md).
|
||||
A. No, it is possible to getting started Kamaji on your laptop with [KinD](./docs/getting-started-with-kamaji.md).
|
||||
|
||||
@@ -1,138 +0,0 @@
|
||||
# Setup a minimal Kamaji for development
|
||||
|
||||
This document explains how to deploy a minimal Kamaji setup on [KinD](https://kind.sigs.k8s.io/) for development scopes. Please refer to the [Kamaji documentation](../../README.md) for understanding all the terms used in this guide, as for example: `admin cluster` and `tenant control plane`.
|
||||
|
||||
## Tools
|
||||
|
||||
We assume you have installed on your workstation:
|
||||
|
||||
- [Docker](https://docs.docker.com/engine/install/)
|
||||
- [KinD](https://kind.sigs.k8s.io/)
|
||||
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
|
||||
- [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
|
||||
- [jq](https://stedolan.github.io/jq/)
|
||||
- [openssl](https://www.openssl.org/)
|
||||
- [cfssl](https://github.com/cloudflare/cfssl)
|
||||
- [cfssljson](https://github.com/cloudflare/cfssl)
|
||||
|
||||
## Setup Kamaji on KinD
|
||||
|
||||
The instance of Kamaji is made of a single node hosting:
|
||||
|
||||
- admin control-plane
|
||||
- admin worker
|
||||
- multi-tenant etcd cluster
|
||||
|
||||
The multi-tenant etcd cluster is deployed as statefulset into the Kamaji node.
|
||||
|
||||
Run `make kamaji` to setup Kamaji on KinD.
|
||||
|
||||
```bash
|
||||
cd ./deploy/kind
|
||||
make kamaji
|
||||
```
|
||||
|
||||
At this moment you will have your KinD up and running and ETCD cluster in multitenant mode.
|
||||
|
||||
### Install Kamaji
|
||||
|
||||
```bash
|
||||
$ kubectl apply -f ../../config/install.yaml
|
||||
```
|
||||
|
||||
### Deploy Tenant Control Plane
|
||||
|
||||
Now it is the moment of deploying your first tenant control plane.
|
||||
|
||||
```bash
|
||||
$ kubectl apply -f - <<EOF
|
||||
apiVersion: kamaji.clastix.io/v1alpha1
|
||||
kind: TenantControlPlane
|
||||
metadata:
|
||||
name: tenant1
|
||||
spec:
|
||||
controlPlane:
|
||||
deployment:
|
||||
replicas: 2
|
||||
additionalMetadata:
|
||||
annotations:
|
||||
environment.clastix.io: tenant1
|
||||
tier.clastix.io: "0"
|
||||
labels:
|
||||
tenant.clastix.io: tenant1
|
||||
kind.clastix.io: deployment
|
||||
service:
|
||||
additionalMetadata:
|
||||
annotations:
|
||||
environment.clastix.io: tenant1
|
||||
tier.clastix.io: "0"
|
||||
labels:
|
||||
tenant.clastix.io: tenant1
|
||||
kind.clastix.io: service
|
||||
serviceType: NodePort
|
||||
ingress:
|
||||
enabled: false
|
||||
kubernetes:
|
||||
version: "v1.23.4"
|
||||
kubelet:
|
||||
cgroupfs: cgroupfs
|
||||
admissionControllers:
|
||||
- LimitRanger
|
||||
- ResourceQuota
|
||||
networkProfile:
|
||||
address: "172.18.0.2"
|
||||
port: 31443
|
||||
domain: "clastix.labs"
|
||||
serviceCidr: "10.96.0.0/16"
|
||||
podCidr: "10.244.0.0/16"
|
||||
dnsServiceIPs:
|
||||
- "10.96.0.10"
|
||||
EOF
|
||||
```
|
||||
|
||||
> Check networkProfile fields according to your installation
|
||||
> To let Kamaji works in kind, you have indicate that the service must be [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport)
|
||||
|
||||
### Get Kubeconfig
|
||||
|
||||
Let's retrieve kubeconfig and store in `/tmp/kubeconfig`
|
||||
|
||||
```bash
|
||||
$ kubectl get secrets tenant1-admin-kubeconfig -o json \
|
||||
| jq -r '.data["admin.conf"]' \
|
||||
| base64 -d > /tmp/kubeconfig
|
||||
```
|
||||
|
||||
It can be export it, to facilitate the next tasks:
|
||||
|
||||
```bash
|
||||
$ export KUBECONFIG=/tmp/kubeconfig
|
||||
```
|
||||
|
||||
### Install CNI
|
||||
|
||||
We highly recommend to install [kindnet](https://github.com/aojea/kindnet) as CNI for your kamaji TCP.
|
||||
|
||||
```bash
|
||||
$ kubectl create -f https://raw.githubusercontent.com/aojea/kindnet/master/install-kindnet.yaml
|
||||
```
|
||||
|
||||
### Join worker nodes
|
||||
|
||||
```bash
|
||||
$ make kamaji-kind-worker-join
|
||||
```
|
||||
|
||||
> To add more worker nodes, run again the command above.
|
||||
|
||||
Check out the node:
|
||||
|
||||
```bash
|
||||
$ kubectl get nodes
|
||||
NAME STATUS ROLES AGE VERSION
|
||||
d2d4b468c9de Ready <none> 44s v1.23.4
|
||||
```
|
||||
|
||||
> For more complex scenarios (exposing port, different version and so on), run `join-node.bash`
|
||||
|
||||
Tenant control plane provision has been finished in a minimal Kamaji setup based on KinD. Therefore, you could develop, test and make your own experiments with Kamaji.
|
||||
@@ -1,604 +1,143 @@
|
||||
# Setup a Kamaji environment
|
||||
This getting started guide will lead you through the process of creating a basic working Kamaji setup.
|
||||
# Setup a minimal Kamaji for development
|
||||
|
||||
Kamaji requires:
|
||||
This document explains how to deploy a minimal Kamaji setup on [KinD](https://kind.sigs.k8s.io/) for development scopes. Please refer to the [Kamaji documentation](../README.md) for understanding all the terms used in this guide, as for example: `admin cluster` and `tenant control plane`.
|
||||
|
||||
- (optional) a bootstrap node;
|
||||
- a multi-tenant `etcd` cluster made of 3 nodes hosting the datastore for the `Tenant`s' clusters
|
||||
- a Kubernetes cluster, running the admin and Tenant Control Planes
|
||||
- an arbitrary number of machines hosting `Tenant`s' workloads
|
||||
## Tools
|
||||
|
||||
> In this guide, we assume all machines are running `Ubuntu 20.04`.
|
||||
We assume you have installed on your workstation:
|
||||
|
||||
* [Prepare the bootstrap workspace](#prepare-the-bootstrap-workspace)
|
||||
* [Access Admin cluster](#access-admin-cluster)
|
||||
* [Setup external multi-tenant etcd](#setup-external-multi-tenant-etcd)
|
||||
* [Setup internal multi-tenant etcd](#setup-internal-multi-tenant-etcd)
|
||||
* [Install Kamaji controller](#install-kamaji-controller)
|
||||
* [Setup Tenant cluster](#setup-tenant-cluster)
|
||||
- [Docker](https://docs.docker.com/engine/install/)
|
||||
- [KinD](https://kind.sigs.k8s.io/)
|
||||
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
|
||||
- [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
|
||||
- [jq](https://stedolan.github.io/jq/)
|
||||
- [openssl](https://www.openssl.org/)
|
||||
- [cfssl](https://github.com/cloudflare/cfssl)
|
||||
- [cfssljson](https://github.com/cloudflare/cfssl)
|
||||
|
||||
## Prepare the bootstrap workspace
|
||||
This getting started guide is supposed to be run from a remote or local bootstrap machine.
|
||||
First, prepare the workspace directory:
|
||||
## Setup Kamaji on KinD
|
||||
|
||||
```
|
||||
git clone https://github.com/clastix/kamaji
|
||||
cd kamaji/deploy
|
||||
```
|
||||
The instance of Kamaji is made of a single node hosting:
|
||||
|
||||
Throughout the instructions, shell variables are used to indicate values that you should adjust to your own environment.
|
||||
- admin control-plane
|
||||
- admin worker
|
||||
- multi-tenant etcd cluster
|
||||
|
||||
### Install required tools
|
||||
On the bootstrap machine, install all the required tools to work with a Kamaji setup.
|
||||
The multi-tenant etcd cluster is deployed as statefulset into the Kamaji node.
|
||||
|
||||
#### cfssl and cfssljson
|
||||
The `cfssl` and `cfssljson` command line utilities will be used in addition to `kubeadm` to provision the PKI Infrastructure and generate TLS certificates.
|
||||
|
||||
```
|
||||
wget -q --show-progress --https-only --timestamping \
|
||||
https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssl \
|
||||
https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssljson
|
||||
|
||||
chmod +x cfssl cfssljson
|
||||
sudo mv cfssl cfssljson /usr/local/bin/
|
||||
```
|
||||
|
||||
#### Kubernetes tools
|
||||
Install `kubeadm` and `kubectl`
|
||||
Run `make kamaji` to setup Kamaji on KinD.
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && \
|
||||
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && \
|
||||
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
|
||||
sudo apt update && sudo apt install -y kubeadm kubectl --allow-change-held-packages && \
|
||||
sudo apt-mark hold kubeadm kubectl
|
||||
cd ./deploy/kind
|
||||
make kamaji
|
||||
```
|
||||
|
||||
#### etcdctl
|
||||
For administration of the `etcd` cluster, download and install the `etcdctl` CLI utility on the bootstrap machine
|
||||
At this moment you will have your KinD up and running and ETCD cluster in multitenant mode.
|
||||
|
||||
### Install Kamaji
|
||||
|
||||
```bash
|
||||
ETCD_VER=v3.5.1
|
||||
ETCD_URL=https://storage.googleapis.com/etcd
|
||||
curl -L ${ETCD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o etcd-${ETCD_VER}-linux-amd64.tar.gz
|
||||
tar xzvf etcd-${ETCD_VER}-linux-amd64.tar.gz etcd-${ETCD_VER}-linux-amd64/etcdctl
|
||||
sudo cp etcd-${ETCD_VER}-linux-amd64/etcdctl /usr/bin/etcdctl
|
||||
rm -rf etcd-${ETCD_VER}-linux-amd64*
|
||||
$ kubectl apply -f ../../config/install.yaml
|
||||
```
|
||||
|
||||
Verify `etcdctl` version is installed
|
||||
### Deploy Tenant Control Plane
|
||||
|
||||
Now it is the moment of deploying your first tenant control plane.
|
||||
|
||||
```bash
|
||||
etcdctl version
|
||||
etcdctl version: 3.5.1
|
||||
API version: 3.5
|
||||
```
|
||||
|
||||
|
||||
## Access Admin cluster
|
||||
In Kamaji, an Admin Cluster is a regular Kubernetes cluster which hosts zero to many Tenant Cluster Control Planes running as pods. The admin cluster acts as management cluster for all the Tenant clusters and implements Monitoring, Logging, and Governance of all the Kamaji setup, including all Tenant clusters.
|
||||
|
||||
Any regular and conformant Kubernetes v1.22+ cluster can be turned into a Kamaji setup. Currently we tested:
|
||||
|
||||
- [Kubernetes installed with `kubeadm`](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/).
|
||||
- [Azure AKS managed service](./kamaji-on-azure.md).
|
||||
- [KinD for local development](../deploy/kind/README.md ).
|
||||
|
||||
The admin cluster should provide:
|
||||
|
||||
- CNI module installed, eg. Calico
|
||||
- Support for LoadBalancer Service Type, eg. MetalLB
|
||||
- Ingress Controller
|
||||
- CSI module installed with StorageClass for multi-tenant `etcd`
|
||||
- Monitoring Stack, eg. Prometheus and Grafana
|
||||
|
||||
Make sure you have a `kubeconfig` file with admin permissions on the cluster you want to turn into a Kamaji Admin Cluster.
|
||||
|
||||
## Setup external multi-tenant etcd
|
||||
In this section, we're going to setup a multi-tenant `etcd` cluster on dedicated nodes. Alternatively, if you want to use an internal `etcd` cluster as Kubernetes StatefulSet, jump [here](#setup-internal-multi-tenant-etcd).
|
||||
|
||||
### Ensure host access
|
||||
From the bootstrap machine load the environment for external `etcd` setup:
|
||||
|
||||
```bash
|
||||
source kamaji-external-etcd.env
|
||||
```
|
||||
|
||||
The installer requires a user that has access to all hosts. In order to run the installer as a non-root user, first configure passwordless sudo rights each host:
|
||||
|
||||
Generate an SSH key on the host you run the installer on:
|
||||
|
||||
```bash
|
||||
ssh-keygen -t rsa
|
||||
```
|
||||
|
||||
> Do not use a password.
|
||||
|
||||
Distribute the key to the other cluster hosts.
|
||||
|
||||
Depending on your environment, use a bash loop:
|
||||
|
||||
```bash
|
||||
HOSTS=(${ETCD0} ${ETCD1} ${ETCD2})
|
||||
for i in "${!HOSTS[@]}"; do
|
||||
HOST=${HOSTS[$i]}
|
||||
ssh-copy-id -i ~/.ssh/id_rsa.pub $HOST;
|
||||
done
|
||||
```
|
||||
|
||||
> Alternatively, inject the generated public key into machines metadata.
|
||||
|
||||
Confirm that you can access each host from bootstrap machine:
|
||||
|
||||
```bash
|
||||
HOSTS=(${ETCD0} ${ETCD1} ${ETCD2})
|
||||
for i in "${!HOSTS[@]}"; do
|
||||
HOST=${HOSTS[$i]}
|
||||
ssh ${USER}@${HOST} -t 'hostname';
|
||||
done
|
||||
```
|
||||
|
||||
### Configure disk layout
|
||||
As per `etcd` [requirements](https://etcd.io/docs/v3.5/op-guide/hardware/#disks), back `etcd`’s storage with a SSD. A SSD usually provides lower write latencies and with less variance than a spinning disk, thus improving the stability and reliability of `etcd`.
|
||||
|
||||
For each `etcd` machine, we assume an additional `sdb` disk of 10GB:
|
||||
|
||||
```
|
||||
clastix@kamaji-etcd-00:~$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
sda 8:0 0 16G 0 disk
|
||||
├─sda1 8:1 0 15.9G 0 part /
|
||||
├─sda14 8:14 0 4M 0 part
|
||||
└─sda15 8:15 0 106M 0 part /boot/efi
|
||||
sdb 8:16 0 10G 0 disk
|
||||
sr0 11:0 1 4M 0 rom
|
||||
```
|
||||
|
||||
Create partition, format, and mount the `etcd` disk, by running the script below from the bootstrap machine:
|
||||
|
||||
> If you already used the `etcd` disks, please make sure to wipe the partitions with `sudo wipefs --all --force /dev/sdb` before to attempt to recreate them.
|
||||
|
||||
```bash
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
ssh ${USER}@${HOST} -t 'echo type=83 | sudo sfdisk -f -q /dev/sdb'
|
||||
ssh ${USER}@${HOST} -t 'sudo mkfs -F -q -t ext4 /dev/sdb1'
|
||||
ssh ${USER}@${HOST} -t 'sudo mkdir -p /var/lib/etcd'
|
||||
ssh ${USER}@${HOST} -t 'sudo e2label /dev/sdb1 ETCD'
|
||||
ssh ${USER}@${HOST} -t 'echo LABEL=ETCD /var/lib/etcd ext4 defaults 0 1 | sudo tee -a /etc/fstab'
|
||||
ssh ${USER}@${HOST} -t 'sudo mount -a'
|
||||
ssh ${USER}@${HOST} -t 'sudo lsblk -f'
|
||||
done
|
||||
```
|
||||
|
||||
### Install prerequisites
|
||||
Use bash script `nodes-prerequisites.sh` to install all the dependencies on all the cluster nodes:
|
||||
|
||||
- Install `containerd` as container runtime
|
||||
- Install `crictl`, the command line for working with `containerd`
|
||||
- Install `kubectl`, `kubelet`, and `kubeadm` in the desired version, eg. `v1.24.0`
|
||||
|
||||
Run the installation script:
|
||||
|
||||
```bash
|
||||
VERSION=v1.24.0
|
||||
./nodes-prerequisites.sh ${VERSION:1} ${HOSTS[@]}
|
||||
```
|
||||
|
||||
### Configure kubelet
|
||||
|
||||
On each `etcd` node, configure the `kubelet` service to start `etcd` static pods using `containerd` as container runtime, by running the script below from the bootstrap machine:
|
||||
|
||||
```bash
|
||||
cat << EOF > 20-etcd-service-manager.conf
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock
|
||||
Restart=always
|
||||
EOF
|
||||
```
|
||||
|
||||
```
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
scp 20-etcd-service-manager.conf ${USER}@${HOST}:
|
||||
ssh ${USER}@${HOST} -t 'sudo chown -R root:root 20-etcd-service-manager.conf && sudo mv 20-etcd-service-manager.conf /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl daemon-reload'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl start kubelet'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl enable kubelet'
|
||||
done
|
||||
|
||||
rm -f 20-etcd-service-manager.conf
|
||||
```
|
||||
|
||||
### Create configuration
|
||||
Create temp directories to store files that will end up on `etcd` hosts:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/${ETCD0}/ /tmp/${ETCD1}/ /tmp/${ETCD2}/
|
||||
NAMES=("etcd00" "etcd01" "etcd02")
|
||||
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
NAME=${NAMES[$i]}
|
||||
|
||||
cat <<EOF | sudo tee /tmp/${HOST}/kubeadmcfg.yaml
|
||||
apiVersion: "kubeadm.k8s.io/v1beta2"
|
||||
kind: ClusterConfiguration
|
||||
etcd:
|
||||
local:
|
||||
serverCertSANs:
|
||||
- "${HOST}"
|
||||
peerCertSANs:
|
||||
- "${HOST}"
|
||||
extraArgs:
|
||||
initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380,${NAMES[2]}=https://${ETCDHOSTS[2]}:2380
|
||||
initial-cluster-state: new
|
||||
name: ${NAME}
|
||||
listen-peer-urls: https://${HOST}:2380
|
||||
listen-client-urls: https://${HOST}:2379
|
||||
advertise-client-urls: https://${HOST}:2379
|
||||
initial-advertise-peer-urls: https://${HOST}:2380
|
||||
auto-compaction-mode: periodic
|
||||
auto-compaction-retention: 5m
|
||||
quota-backend-bytes: '8589934592'
|
||||
EOF
|
||||
done
|
||||
```
|
||||
> Note:
|
||||
>
|
||||
> ##### Etcd compaction
|
||||
>
|
||||
> By enabling `etcd` authentication, it prevents the tenant apiservers (clients of `etcd`) to issue compaction requests. We set `etcd` to automatically compact the keyspace with the `--auto-compaction-*` option with a period of hours or minutes. When `--auto-compaction-mode=periodic` and `--auto-compaction-retention=5m` and writes per minute are about 1000, `etcd` compacts revision 5000 for every 5 minute.
|
||||
>
|
||||
> ##### Etcd storage quota
|
||||
>
|
||||
> Currently, `etcd` is limited in storage size, defaulted to `2GB` and configurable with `--quota-backend-bytes` flag up to `8GB`. In Kamaji, we use a single `etcd` to store multiple tenant clusters, so we need to increase this size. Please, note `etcd` warns at startup if the configured value exceeds `8GB`.
|
||||
|
||||
### Generate certificates
|
||||
On the bootstrap machine, using `kubeadm` init phase, create and distribute `etcd` CA certificates:
|
||||
|
||||
```bash
|
||||
sudo kubeadm init phase certs etcd-ca
|
||||
mkdir kamaji
|
||||
sudo cp -r /etc/kubernetes/pki/etcd kamaji
|
||||
sudo chown -R ${USER}. kamaji/etcd
|
||||
```
|
||||
|
||||
For each `etcd` host:
|
||||
|
||||
```bash
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
sudo kubeadm init phase certs etcd-server --config=/tmp/${HOST}/kubeadmcfg.yaml
|
||||
sudo kubeadm init phase certs etcd-peer --config=/tmp/${HOST}/kubeadmcfg.yaml
|
||||
sudo kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST}/kubeadmcfg.yaml
|
||||
sudo cp -R /etc/kubernetes/pki /tmp/${HOST}/
|
||||
sudo find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
|
||||
done
|
||||
```
|
||||
|
||||
### Startup the cluster
|
||||
Upload certificates on each `etcd` node and restart the `kubelet`
|
||||
|
||||
```bash
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
sudo chown -R ${USER}. /tmp/${HOST}
|
||||
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
|
||||
ssh ${USER}@${HOST} -t 'sudo chown -R root:root pki'
|
||||
ssh ${USER}@${HOST} -t 'sudo mv pki /etc/kubernetes/'
|
||||
ssh ${USER}@${HOST} -t 'sudo kubeadm init phase etcd local --config=kubeadmcfg.yaml'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl daemon-reload'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl restart kubelet'
|
||||
done
|
||||
```
|
||||
|
||||
This will start the static `etcd` pod on each node and then the cluster gets formed.
|
||||
|
||||
Generate certificates for the `root` user
|
||||
|
||||
```bash
|
||||
cat > root-csr.json <<EOF
|
||||
{
|
||||
"CN": "root",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
```bash
|
||||
cfssl gencert \
|
||||
-ca=kamaji/etcd/ca.crt \
|
||||
-ca-key=kamaji/etcd/ca.key \
|
||||
-config=cfssl-cert-config.json \
|
||||
-profile=client-authentication \
|
||||
root-csr.json | cfssljson -bare root
|
||||
```
|
||||
|
||||
```bash
|
||||
cp root.pem kamaji/etcd/root.crt
|
||||
cp root-key.pem kamaji/etcd/root.key
|
||||
rm root*
|
||||
```
|
||||
|
||||
The result should be:
|
||||
|
||||
```bash
|
||||
$ tree kamaji
|
||||
kamaji
|
||||
└── etcd
|
||||
├── ca.crt
|
||||
├── ca.key
|
||||
├── root.crt
|
||||
└── root.key
|
||||
```
|
||||
|
||||
Use the `root` user to check the just formed `etcd` cluster is in health state
|
||||
|
||||
```bash
|
||||
export ETCDCTL_CACERT=kamaji/etcd/ca.crt
|
||||
export ETCDCTL_CERT=kamaji/etcd/root.crt
|
||||
export ETCDCTL_KEY=kamaji/etcd/root.key
|
||||
export ETCDCTL_ENDPOINTS=https://${ETCD0}:2379
|
||||
|
||||
etcdctl member list -w table
|
||||
```
|
||||
|
||||
The result should be something like this:
|
||||
|
||||
```
|
||||
+------------------+---------+--------+----------------------------+----------------------------+------------+
|
||||
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
|
||||
+------------------+---------+--------+----------------------------+----------------------------+------------+
|
||||
| 72657d6307364226 | started | etcd01 | https://192.168.32.11:2380 | https://192.168.32.11:2379 | false |
|
||||
| 91eb892c5ee87610 | started | etcd00 | https://192.168.32.10:2380 | https://192.168.32.10:2379 | false |
|
||||
| e9971c576949c34e | started | etcd02 | https://192.168.32.12:2380 | https://192.168.32.12:2379 | false |
|
||||
+------------------+---------+--------+----------------------------+----------------------------+------------+
|
||||
```
|
||||
|
||||
### Enable multi-tenancy
|
||||
The `root` user has full access to `etcd`, must be created before activating authentication. The `root` user must have the `root` role and is allowed to change anything inside `etcd`.
|
||||
|
||||
```bash
|
||||
etcdctl user add --no-password=true root
|
||||
etcdctl role add root
|
||||
etcdctl user grant-role root root
|
||||
etcdctl auth enable
|
||||
```
|
||||
|
||||
### Cleanup
|
||||
If you want to get rid of the etcd cluster, for each node, login and clean it:
|
||||
|
||||
```bash
|
||||
HOSTS=(${ETCD0} ${ETCD1} ${ETCD2})
|
||||
for i in "${!HOSTS[@]}"; do
|
||||
HOST=${HOSTS[$i]}
|
||||
ssh ${USER}@${HOST} -t 'sudo kubeadm reset -f';
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl reboot';
|
||||
done
|
||||
```
|
||||
|
||||
## Setup internal multi-tenant etcd
|
||||
If you opted for an internal etcd cluster running in the Kamaji admin cluster, follow steps below.
|
||||
|
||||
From the bootstrap machine load the environment for internal `etcd` setup:
|
||||
|
||||
```bash
|
||||
source kamaji-internal-etcd.env
|
||||
```
|
||||
|
||||
### Generate certificates
|
||||
On the bootstrap machine, using `kubeadm` init phase, create the `etcd` CA certificates:
|
||||
|
||||
```bash
|
||||
sudo kubeadm init phase certs etcd-ca
|
||||
mkdir kamaji
|
||||
sudo cp -r /etc/kubernetes/pki/etcd kamaji
|
||||
sudo chown -R ${USER}. kamaji/etcd
|
||||
```
|
||||
|
||||
Generate the `etcd` certificates for peers:
|
||||
|
||||
```
|
||||
cat << EOF | tee kamaji/etcd/peer-csr.json
|
||||
{
|
||||
"CN": "etcd",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
},
|
||||
"hosts": [
|
||||
"127.0.0.1",
|
||||
"etcd-0",
|
||||
"etcd-0.etcd",
|
||||
"etcd-0.etcd.${ETCD_NAMESPACE}.svc",
|
||||
"etcd-0.etcd.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-1",
|
||||
"etcd-1.etcd",
|
||||
"etcd-1.etcd.${ETCD_NAMESPACE}.svc",
|
||||
"etcd-1.etcd.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-2",
|
||||
"etcd-2.etcd",
|
||||
"etcd-2.etcd.${ETCD_NAMESPACE}.svc",
|
||||
"etcd-2.etcd.${ETCD_NAMESPACE}.cluster.local"
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
cfssl gencert -ca=kamaji/etcd/ca.crt -ca-key=kamaji/etcd/ca.key \
|
||||
-config=cfssl-cert-config.json \
|
||||
-profile=peer-authentication kamaji/etcd/peer-csr.json | cfssljson -bare kamaji/etcd/peer
|
||||
|
||||
```
|
||||
|
||||
Generate the `etcd` certificates for server:
|
||||
|
||||
```
|
||||
cat << EOF | tee kamaji/etcd/server-csr.json
|
||||
{
|
||||
"CN": "etcd",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
},
|
||||
"hosts": [
|
||||
"127.0.0.1",
|
||||
"etcd-server",
|
||||
"etcd-server.${ETCD_NAMESPACE}.svc",
|
||||
"etcd-server.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-0.etcd.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-1.etcd.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-2.etcd.${ETCD_NAMESPACE}.svc.cluster.local"
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
cfssl gencert -ca=kamaji/etcd/ca.crt -ca-key=kamaji/etcd/ca.key \
|
||||
-config=cfssl-cert-config.json \
|
||||
-profile=peer-authentication kamaji/etcd/server-csr.json | cfssljson -bare kamaji/etcd/server
|
||||
```
|
||||
|
||||
Generate certificates for the `root` user of the `etcd`
|
||||
|
||||
```
|
||||
cat << EOF | tee kamaji/etcd/root-csr.json
|
||||
{
|
||||
"CN": "root",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
}
|
||||
}
|
||||
EOF
|
||||
|
||||
cfssl gencert -ca=kamaji/etcd/ca.crt -ca-key=kamaji/etcd/ca.key \
|
||||
-config=cfssl-cert-config.json \
|
||||
-profile=client-authentication kamaji/etcd/root-csr.json | cfssljson -bare kamaji/etcd/root
|
||||
```
|
||||
|
||||
Install the `etcd` in the Kamaji admin cluster
|
||||
|
||||
```bash
|
||||
kubectl create namespace ${ETCD_NAMESPACE}
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} create secret generic etcd-certs \
|
||||
--from-file=kamaji/etcd/ca.crt \
|
||||
--from-file=kamaji/etcd/ca.key \
|
||||
--from-file=kamaji/etcd/peer-key.pem --from-file=kamaji/etcd/peer.pem \
|
||||
--from-file=kamaji/etcd/server-key.pem --from-file=kamaji/etcd/server.pem
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} apply -f etcd/etcd-cluster.yaml
|
||||
```
|
||||
|
||||
Install an `etcd` client to interact with the `etcd` server
|
||||
|
||||
```bash
|
||||
kubectl -n ${ETCD_NAMESPACE} create secret tls root-certs \
|
||||
--key=kamaji/etcd/root-key.pem \
|
||||
--cert=kamaji/etcd/root.pem
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} apply -f etcd/etcd-client.yaml
|
||||
```
|
||||
|
||||
Wait the etcd instances discover each other and the cluster is formed:
|
||||
|
||||
```bash
|
||||
kubectl -n ${ETCD_NAMESPACE} wait pod --for=condition=ready -l app=etcd --timeout=120s
|
||||
echo -n "\nChecking endpoint's health..."
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- /bin/bash -c "etcdctl endpoint health 1>/dev/null 2>/dev/null; until [ \$$? -eq 0 ]; do sleep 10; printf "."; etcdctl endpoint health 1>/dev/null 2>/dev/null; done;"
|
||||
echo -n "\netcd cluster's health:\n"
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- /bin/bash -c "etcdctl endpoint health"
|
||||
echo -n "\nWaiting for all members..."
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- /bin/bash -c "until [ \$$(etcdctl member list 2>/dev/null | wc -l) -eq 3 ]; do sleep 10; printf '.'; done;"
|
||||
@echo -n "\netcd's members:\n"
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- /bin/bash -c "etcdctl member list -w table"
|
||||
```
|
||||
|
||||
### Enable multi-tenancy
|
||||
The `root` user has full access to `etcd`, must be created before activating authentication. The `root` user must have the `root` role and is allowed to change anything inside `etcd`.
|
||||
|
||||
```bash
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- etcdctl user add --no-password=true root
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- etcdctl role add root
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- etcdctl user grant-role root root
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- etcdctl auth enable
|
||||
```
|
||||
|
||||
|
||||
## Install Kamaji controller
|
||||
Currently, the behaviour of the Kamaji controller for Tenant Control Plane is controlled by (in this order):
|
||||
|
||||
- CLI flags
|
||||
- Environment variables
|
||||
- Configuration file `kamaji.yaml` built into the image
|
||||
|
||||
By default Kamaji search for the configuration file and uses parameters found inside of it. In case some environment variable are passed, this will override configuration file parameters. In the end, if also a CLI flag is passed, this will override both env vars and config file as well.
|
||||
|
||||
There are multiple ways to deploy the Kamaji controller:
|
||||
|
||||
- Use the single YAML file installer
|
||||
- Use Kustomize with Makefile
|
||||
- Use the Kamaji Helm Chart
|
||||
|
||||
The Kamaji controller needs to access the multi-tenant `etcd` in order to provision the access for tenant `kube-apiserver`.
|
||||
|
||||
Create the secrets containing the `etcd` certificates
|
||||
|
||||
```bash
|
||||
kubectl create namespace kamaji-system
|
||||
kubectl -n kamaji-system create secret generic etcd-certs \
|
||||
--from-file=kamaji/etcd/ca.crt \
|
||||
--from-file=kamaji/etcd/ca.key
|
||||
|
||||
kubectl -n kamaji-system create secret tls root-client-certs \
|
||||
--cert=kamaji/etcd/root.crt \
|
||||
--key=kamaji/etcd/root.key
|
||||
```
|
||||
|
||||
### Install with a single manifest
|
||||
Install with the single YAML file installer:
|
||||
|
||||
```bash
|
||||
kubectl -n kamaji-system apply -f ../config/install.yaml
|
||||
```
|
||||
|
||||
Make sure to patch the `etcd` endpoints of the Kamaji controller, according to your environment:
|
||||
|
||||
```bash
|
||||
cat > patch-deploy.yaml <<EOF
|
||||
$ kubectl apply -f - <<EOF
|
||||
apiVersion: kamaji.clastix.io/v1alpha1
|
||||
kind: TenantControlPlane
|
||||
metadata:
|
||||
name: tenant1
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: manager
|
||||
args:
|
||||
- --health-probe-bind-address=:8081
|
||||
- --metrics-bind-address=127.0.0.1:8080
|
||||
- --leader-elect
|
||||
- --etcd-endpoints=${ETCD0}:2379,${ETCD1}:2379,${ETCD2}:2379
|
||||
controlPlane:
|
||||
deployment:
|
||||
replicas: 2
|
||||
additionalMetadata:
|
||||
annotations:
|
||||
environment.clastix.io: tenant1
|
||||
tier.clastix.io: "0"
|
||||
labels:
|
||||
tenant.clastix.io: tenant1
|
||||
kind.clastix.io: deployment
|
||||
service:
|
||||
additionalMetadata:
|
||||
annotations:
|
||||
environment.clastix.io: tenant1
|
||||
tier.clastix.io: "0"
|
||||
labels:
|
||||
tenant.clastix.io: tenant1
|
||||
kind.clastix.io: service
|
||||
serviceType: NodePort
|
||||
ingress:
|
||||
enabled: false
|
||||
kubernetes:
|
||||
version: "v1.23.4"
|
||||
kubelet:
|
||||
cgroupfs: cgroupfs
|
||||
admissionControllers:
|
||||
- LimitRanger
|
||||
- ResourceQuota
|
||||
networkProfile:
|
||||
address: "172.18.0.2"
|
||||
port: 31443
|
||||
domain: "clastix.labs"
|
||||
serviceCidr: "10.96.0.0/16"
|
||||
podCidr: "10.244.0.0/16"
|
||||
dnsServiceIPs:
|
||||
- "10.96.0.10"
|
||||
addons:
|
||||
coreDNS:
|
||||
enabled: true
|
||||
kubeProxy:
|
||||
enabled: true
|
||||
EOF
|
||||
|
||||
kubectl -n kamaji-system patch \
|
||||
deployment kamaji-controller-manager \
|
||||
--patch-file patch-deploy.yaml
|
||||
```
|
||||
|
||||
The Kamaji Tenant Control Plane controller is now running on the Admin Cluster:
|
||||
> Check networkProfile fields according to your installation
|
||||
> To let Kamaji works in kind, you have indicate that the service must be [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport)
|
||||
|
||||
### Get Kubeconfig
|
||||
|
||||
Let's retrieve kubeconfig and store in `/tmp/kubeconfig`
|
||||
|
||||
```bash
|
||||
kubectl -n kamaji-system get deploy
|
||||
NAME READY UP-TO-DATE AVAILABLE AGE
|
||||
operator-controller-manager 1/1 1 1 14h
|
||||
$ kubectl get secrets tenant1-admin-kubeconfig -o json \
|
||||
| jq -r '.data["admin.conf"]' \
|
||||
| base64 -d > /tmp/kubeconfig
|
||||
```
|
||||
|
||||
It can be export it, to facilitate the next tasks:
|
||||
|
||||
```bash
|
||||
$ export KUBECONFIG=/tmp/kubeconfig
|
||||
```
|
||||
|
||||
## Setup Tenant Cluster
|
||||
Now you are getting an Admin Cluster available to run multiple Tenant Control Planes, deployed by the Kamaji controller. Please, refer to the Kamaji Tenant Deployment [guide](./kamaji-tenant-deployment-guide.md).
|
||||
### Install CNI
|
||||
|
||||
We highly recommend to install [kindnet](https://github.com/aojea/kindnet) as CNI for your kamaji TCP.
|
||||
|
||||
```bash
|
||||
$ kubectl create -f https://raw.githubusercontent.com/aojea/kindnet/master/install-kindnet.yaml
|
||||
```
|
||||
|
||||
### Join worker nodes
|
||||
|
||||
```bash
|
||||
$ make kamaji-kind-worker-join
|
||||
```
|
||||
|
||||
> To add more worker nodes, run again the command above.
|
||||
|
||||
Check out the node:
|
||||
|
||||
```bash
|
||||
$ kubectl get nodes
|
||||
NAME STATUS ROLES AGE VERSION
|
||||
d2d4b468c9de Ready <none> 44s v1.23.4
|
||||
```
|
||||
|
||||
> For more complex scenarios (exposing port, different version and so on), run `join-node.bash`
|
||||
|
||||
Tenant control plane provision has been finished in a minimal Kamaji setup based on KinD. Therefore, you could develop, test and make your own experiments with Kamaji.
|
||||
|
||||
@@ -82,21 +82,10 @@ source kamaji-tenant-azure.env
|
||||
### On Kamaji side
|
||||
With Kamaji on AKS, the tenant control plane is accessible:
|
||||
|
||||
- from tenant work nodes through an internal loadbalancer as `https://${TENANT_ADDR}:${TENANT_PORT}`
|
||||
- from tenant work nodes through an internal loadbalancer as `https://${TENANT_ADDR}:6443`
|
||||
- from tenant admin user through an external loadbalancer `https://${TENANT_NAME}.${KAMAJI_REGION}.cloudapp.azure.com:443`
|
||||
|
||||
#### Allocate an internal IP address for the Tenant Control Plane
|
||||
Currently, Kamaji has a known limitation, meaning the address `${TENANT_ADDR}:${TENANT_PORT}` must be known in advance before to create the Tenant Control Plane. Given this limitation, let's to reserve an IP address and port in the same virtual subnet used by the Kamaji admin cluster:
|
||||
|
||||
|
||||
```bash
|
||||
export TENANT_ADDR=10.240.0.100
|
||||
export TENANT_PORT=6443
|
||||
export TENANT_DOMAIN=$KAMAJI_REGION.cloudapp.azure.com
|
||||
```
|
||||
|
||||
> Make sure the `TENANT_ADDR` value does not overlap with already allocated IP addresses in the AKS virtual network. In the future, Kamaji will implement a dynamic IP allocation.
|
||||
|
||||
Where `TENANT_ADDR` is the Azure internal IP address assigned to the LoadBalancer service created by Kamaji to expose the Tenant Control Plane endpoint.
|
||||
|
||||
#### Create the Tenant Control Plane
|
||||
|
||||
@@ -139,13 +128,17 @@ spec:
|
||||
- ResourceQuota
|
||||
- LimitRanger
|
||||
networkProfile:
|
||||
address: ${TENANT_ADDR}
|
||||
port: ${TENANT_PORT}
|
||||
domain: ${TENANT_DOMAIN}
|
||||
port: 6443
|
||||
domain: ${KAMAJI_REGION}.cloudapp.azure.com
|
||||
serviceCidr: ${TENANT_SVC_CIDR}
|
||||
podCidr: ${TENANT_POD_CIDR}
|
||||
dnsServiceIPs:
|
||||
- ${TENANT_DNS_SERVICE}
|
||||
addons:
|
||||
coreDNS:
|
||||
enabled: true
|
||||
kubeProxy:
|
||||
enabled: true
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
@@ -195,6 +188,11 @@ NAME READY UP-TO-DATE AVAILABLE AGE
|
||||
tenant-00 2/2 2 2 47m
|
||||
```
|
||||
|
||||
Collect the internal IP address of Azure loadbalancer where the Tenant control Plane is exposed:
|
||||
|
||||
```bash
|
||||
TENANT_ADDR=$(kubectl -n ${TENANT_NAMESPACE} get svc ${TENANT_NAME} -o json | jq -r ."status.loadBalancer.ingress[].ip")
|
||||
```
|
||||
|
||||
#### Working with Tenant Control Plane
|
||||
Check the access to the Tenant Control Plane:
|
||||
@@ -225,7 +223,7 @@ NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AG
|
||||
default kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 6m
|
||||
```
|
||||
|
||||
Check out how the Tenant Control Plane advertises itself to workloads:
|
||||
Check out how the Tenant Control Plane advertises itself:
|
||||
|
||||
```
|
||||
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get ep
|
||||
@@ -234,7 +232,7 @@ NAME ENDPOINTS AGE
|
||||
kubernetes 10.240.0.100:6443 57m
|
||||
```
|
||||
|
||||
Make sure it's `${TENANT_ADDR}:${TENANT_PORT}`.
|
||||
Make sure it's `${TENANT_ADDR}:6443`.
|
||||
|
||||
### Prepare the Infrastructure for the Tenant virtual machines
|
||||
Kamaji provides Control Plane as a Service, so the tenant user can join his own virtual machines as worker nodes. Each tenant can place his virtual machines in a dedicated Azure virtual network.
|
||||
@@ -332,7 +330,7 @@ az vmss scale \
|
||||
The current approach for joining nodes is to use the `kubeadm` one therefore, we will create a bootstrap token to perform the action:
|
||||
|
||||
```bash
|
||||
JOIN_CMD=$(echo "sudo kubeadm join ${TENANT_ADDR}:${TENANT_PORT} ")$(kubeadm --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig token create --print-join-command |cut -d" " -f4-)
|
||||
JOIN_CMD=$(echo "sudo kubeadm join ${TENANT_ADDR}:6443 ")$(kubeadm --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig token create --print-join-command |cut -d" " -f4-)
|
||||
```
|
||||
|
||||
A bash loop will be used to join all the available nodes.
|
||||
@@ -398,6 +396,16 @@ kamaji-tenant-worker-02 Ready <none> 10m v1.23.4
|
||||
|
||||
|
||||
## Cleanup
|
||||
To get rid of the Tenant infrastructure, remove the RESOURCE_GROUP: `az group delete --name $TENANT_RG --yes --no-wait`.
|
||||
To get rid of the Tenant infrastructure, remove the RESOURCE_GROUP:
|
||||
|
||||
To get rid of the Kamaji infrastructure, remove the RESOURCE_GROUP: `az group delete --name $KAMAJI_RG --yes --no-wait`.
|
||||
```
|
||||
az group delete --name $TENANT_RG --yes --no-wait
|
||||
```
|
||||
|
||||
To get rid of the Kamaji infrastructure, remove the RESOURCE_GROUP:
|
||||
|
||||
```
|
||||
az group delete --name $KAMAJI_RG --yes --no-wait
|
||||
```
|
||||
|
||||
That's all folks!
|
||||
603
docs/kamaji-deployment-guide.md
Normal file
603
docs/kamaji-deployment-guide.md
Normal file
@@ -0,0 +1,603 @@
|
||||
# Install a Kamaji environment
|
||||
This guide will lead you through the process of creating a basic working Kamaji setup.
|
||||
|
||||
Kamaji requires:
|
||||
|
||||
- (optional) a bootstrap node;
|
||||
- a multi-tenant `etcd` cluster made of 3 nodes hosting the datastore for the `Tenant`s' clusters
|
||||
- a Kubernetes cluster, running the admin and Tenant Control Planes
|
||||
- an arbitrary number of machines hosting `Tenant`s' workloads
|
||||
|
||||
> In this guide, we assume all machines are running `Ubuntu 20.04`.
|
||||
|
||||
* [Prepare the bootstrap workspace](#prepare-the-bootstrap-workspace)
|
||||
* [Access Admin cluster](#access-admin-cluster)
|
||||
* [Setup external multi-tenant etcd](#setup-external-multi-tenant-etcd)
|
||||
* [Setup internal multi-tenant etcd](#setup-internal-multi-tenant-etcd)
|
||||
* [Install Kamaji controller](#install-kamaji-controller)
|
||||
* [Setup Tenant cluster](#setup-tenant-cluster)
|
||||
|
||||
## Prepare the bootstrap workspace
|
||||
This guide is supposed to be run from a remote or local bootstrap machine.
|
||||
First, prepare the workspace directory:
|
||||
|
||||
```
|
||||
git clone https://github.com/clastix/kamaji
|
||||
cd kamaji/deploy
|
||||
```
|
||||
|
||||
Throughout the instructions, shell variables are used to indicate values that you should adjust to your own environment.
|
||||
|
||||
### Install required tools
|
||||
On the bootstrap machine, install all the required tools to work with a Kamaji setup.
|
||||
|
||||
#### cfssl and cfssljson
|
||||
The `cfssl` and `cfssljson` command line utilities will be used in addition to `kubeadm` to provision the PKI Infrastructure and generate TLS certificates.
|
||||
|
||||
```
|
||||
wget -q --show-progress --https-only --timestamping \
|
||||
https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssl \
|
||||
https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssljson
|
||||
|
||||
chmod +x cfssl cfssljson
|
||||
sudo mv cfssl cfssljson /usr/local/bin/
|
||||
```
|
||||
|
||||
#### Kubernetes tools
|
||||
Install `kubeadm` and `kubectl`
|
||||
|
||||
```bash
|
||||
sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && \
|
||||
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && \
|
||||
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
|
||||
sudo apt update && sudo apt install -y kubeadm kubectl --allow-change-held-packages && \
|
||||
sudo apt-mark hold kubeadm kubectl
|
||||
```
|
||||
|
||||
#### etcdctl
|
||||
For administration of the `etcd` cluster, download and install the `etcdctl` CLI utility on the bootstrap machine
|
||||
|
||||
```bash
|
||||
ETCD_VER=v3.5.1
|
||||
ETCD_URL=https://storage.googleapis.com/etcd
|
||||
curl -L ${ETCD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o etcd-${ETCD_VER}-linux-amd64.tar.gz
|
||||
tar xzvf etcd-${ETCD_VER}-linux-amd64.tar.gz etcd-${ETCD_VER}-linux-amd64/etcdctl
|
||||
sudo cp etcd-${ETCD_VER}-linux-amd64/etcdctl /usr/bin/etcdctl
|
||||
rm -rf etcd-${ETCD_VER}-linux-amd64*
|
||||
```
|
||||
|
||||
Verify `etcdctl` version is installed
|
||||
|
||||
```bash
|
||||
etcdctl version
|
||||
etcdctl version: 3.5.1
|
||||
API version: 3.5
|
||||
```
|
||||
|
||||
|
||||
## Access Admin cluster
|
||||
In Kamaji, an Admin Cluster is a regular Kubernetes cluster which hosts zero to many Tenant Cluster Control Planes running as pods. The admin cluster acts as management cluster for all the Tenant clusters and implements Monitoring, Logging, and Governance of all the Kamaji setup, including all Tenant clusters.
|
||||
|
||||
Any regular and conformant Kubernetes v1.22+ cluster can be turned into a Kamaji setup. Currently we tested:
|
||||
|
||||
- [Kubernetes installed with `kubeadm`](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/).
|
||||
- [Azure AKS managed service](./kamaji-on-azure.md).
|
||||
- [KinD for local development](./getting-started-with-kamaji.md ).
|
||||
|
||||
The admin cluster should provide:
|
||||
|
||||
- CNI module installed, eg. Calico
|
||||
- Support for LoadBalancer Service Type, eg. MetalLB or, alternatively, an Ingress Controller
|
||||
- CSI module installed with StorageClass for multi-tenant `etcd`
|
||||
- Monitoring Stack, eg. Prometheus and Grafana
|
||||
|
||||
Make sure you have a `kubeconfig` file with admin permissions on the cluster you want to turn into Kamaji Admin Cluster.
|
||||
|
||||
## Setup external multi-tenant etcd
|
||||
In this section, we're going to setup a multi-tenant `etcd` cluster on dedicated nodes. Alternatively, if you want to use an internal `etcd` cluster as Kubernetes StatefulSet, jump [here](#setup-internal-multi-tenant-etcd).
|
||||
|
||||
### Ensure host access
|
||||
From the bootstrap machine load the environment for external `etcd` setup:
|
||||
|
||||
```bash
|
||||
source kamaji-external-etcd.env
|
||||
```
|
||||
|
||||
The installer requires a user that has access to all hosts. In order to run the installer as a non-root user, first configure passwordless sudo rights each host:
|
||||
|
||||
Generate an SSH key on the host you run the installer on:
|
||||
|
||||
```bash
|
||||
ssh-keygen -t rsa
|
||||
```
|
||||
|
||||
> Do not use a password.
|
||||
|
||||
Distribute the key to the other cluster hosts.
|
||||
|
||||
Depending on your environment, use a bash loop:
|
||||
|
||||
```bash
|
||||
HOSTS=(${ETCD0} ${ETCD1} ${ETCD2})
|
||||
for i in "${!HOSTS[@]}"; do
|
||||
HOST=${HOSTS[$i]}
|
||||
ssh-copy-id -i ~/.ssh/id_rsa.pub $HOST;
|
||||
done
|
||||
```
|
||||
|
||||
> Alternatively, inject the generated public key into machines metadata.
|
||||
|
||||
Confirm that you can access each host from bootstrap machine:
|
||||
|
||||
```bash
|
||||
HOSTS=(${ETCD0} ${ETCD1} ${ETCD2})
|
||||
for i in "${!HOSTS[@]}"; do
|
||||
HOST=${HOSTS[$i]}
|
||||
ssh ${USER}@${HOST} -t 'hostname';
|
||||
done
|
||||
```
|
||||
|
||||
### Configure disk layout
|
||||
As per `etcd` [requirements](https://etcd.io/docs/v3.5/op-guide/hardware/#disks), back `etcd`’s storage with a SSD. A SSD usually provides lower write latencies and with less variance than a spinning disk, thus improving the stability and reliability of `etcd`.
|
||||
|
||||
For each `etcd` machine, we assume an additional `sdb` disk of 10GB:
|
||||
|
||||
```
|
||||
clastix@kamaji-etcd-00:~$ lsblk
|
||||
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
||||
sda 8:0 0 16G 0 disk
|
||||
├─sda1 8:1 0 15.9G 0 part /
|
||||
├─sda14 8:14 0 4M 0 part
|
||||
└─sda15 8:15 0 106M 0 part /boot/efi
|
||||
sdb 8:16 0 10G 0 disk
|
||||
sr0 11:0 1 4M 0 rom
|
||||
```
|
||||
|
||||
Create partition, format, and mount the `etcd` disk, by running the script below from the bootstrap machine:
|
||||
|
||||
> If you already used the `etcd` disks, please make sure to wipe the partitions with `sudo wipefs --all --force /dev/sdb` before to attempt to recreate them.
|
||||
|
||||
```bash
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
ssh ${USER}@${HOST} -t 'echo type=83 | sudo sfdisk -f -q /dev/sdb'
|
||||
ssh ${USER}@${HOST} -t 'sudo mkfs -F -q -t ext4 /dev/sdb1'
|
||||
ssh ${USER}@${HOST} -t 'sudo mkdir -p /var/lib/etcd'
|
||||
ssh ${USER}@${HOST} -t 'sudo e2label /dev/sdb1 ETCD'
|
||||
ssh ${USER}@${HOST} -t 'echo LABEL=ETCD /var/lib/etcd ext4 defaults 0 1 | sudo tee -a /etc/fstab'
|
||||
ssh ${USER}@${HOST} -t 'sudo mount -a'
|
||||
ssh ${USER}@${HOST} -t 'sudo lsblk -f'
|
||||
done
|
||||
```
|
||||
|
||||
### Install prerequisites
|
||||
Use bash script `nodes-prerequisites.sh` to install all the dependencies on all the cluster nodes:
|
||||
|
||||
- Install `containerd` as container runtime
|
||||
- Install `crictl`, the command line for working with `containerd`
|
||||
- Install `kubectl`, `kubelet`, and `kubeadm` in the desired version, eg. `v1.24.0`
|
||||
|
||||
Run the installation script:
|
||||
|
||||
```bash
|
||||
VERSION=v1.24.0
|
||||
./nodes-prerequisites.sh ${VERSION:1} ${HOSTS[@]}
|
||||
```
|
||||
|
||||
### Configure kubelet
|
||||
|
||||
On each `etcd` node, configure the `kubelet` service to start `etcd` static pods using `containerd` as container runtime, by running the script below from the bootstrap machine:
|
||||
|
||||
```bash
|
||||
cat << EOF > 20-etcd-service-manager.conf
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock
|
||||
Restart=always
|
||||
EOF
|
||||
```
|
||||
|
||||
```
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
scp 20-etcd-service-manager.conf ${USER}@${HOST}:
|
||||
ssh ${USER}@${HOST} -t 'sudo chown -R root:root 20-etcd-service-manager.conf && sudo mv 20-etcd-service-manager.conf /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl daemon-reload'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl start kubelet'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl enable kubelet'
|
||||
done
|
||||
|
||||
rm -f 20-etcd-service-manager.conf
|
||||
```
|
||||
|
||||
### Create configuration
|
||||
Create temp directories to store files that will end up on `etcd` hosts:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/${ETCD0}/ /tmp/${ETCD1}/ /tmp/${ETCD2}/
|
||||
NAMES=("etcd00" "etcd01" "etcd02")
|
||||
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
NAME=${NAMES[$i]}
|
||||
|
||||
cat <<EOF | sudo tee /tmp/${HOST}/kubeadmcfg.yaml
|
||||
apiVersion: "kubeadm.k8s.io/v1beta2"
|
||||
kind: ClusterConfiguration
|
||||
etcd:
|
||||
local:
|
||||
serverCertSANs:
|
||||
- "${HOST}"
|
||||
peerCertSANs:
|
||||
- "${HOST}"
|
||||
extraArgs:
|
||||
initial-cluster: ${NAMES[0]}=https://${ETCDHOSTS[0]}:2380,${NAMES[1]}=https://${ETCDHOSTS[1]}:2380,${NAMES[2]}=https://${ETCDHOSTS[2]}:2380
|
||||
initial-cluster-state: new
|
||||
name: ${NAME}
|
||||
listen-peer-urls: https://${HOST}:2380
|
||||
listen-client-urls: https://${HOST}:2379
|
||||
advertise-client-urls: https://${HOST}:2379
|
||||
initial-advertise-peer-urls: https://${HOST}:2380
|
||||
auto-compaction-mode: periodic
|
||||
auto-compaction-retention: 5m
|
||||
quota-backend-bytes: '8589934592'
|
||||
EOF
|
||||
done
|
||||
```
|
||||
> Note:
|
||||
>
|
||||
> ##### Etcd compaction
|
||||
>
|
||||
> By enabling `etcd` authentication, it prevents the tenant apiservers (clients of `etcd`) to issue compaction requests. We set `etcd` to automatically compact the keyspace with the `--auto-compaction-*` option with a period of hours or minutes. When `--auto-compaction-mode=periodic` and `--auto-compaction-retention=5m` and writes per minute are about 1000, `etcd` compacts revision 5000 for every 5 minute.
|
||||
>
|
||||
> ##### Etcd storage quota
|
||||
>
|
||||
> Currently, `etcd` is limited in storage size, defaulted to `2GB` and configurable with `--quota-backend-bytes` flag up to `8GB`. In Kamaji, we use a single `etcd` to store multiple tenant clusters, so we need to increase this size. Please, note `etcd` warns at startup if the configured value exceeds `8GB`.
|
||||
|
||||
### Generate certificates
|
||||
On the bootstrap machine, using `kubeadm` init phase, create and distribute `etcd` CA certificates:
|
||||
|
||||
```bash
|
||||
sudo kubeadm init phase certs etcd-ca
|
||||
mkdir kamaji
|
||||
sudo cp -r /etc/kubernetes/pki/etcd kamaji
|
||||
sudo chown -R ${USER}. kamaji/etcd
|
||||
```
|
||||
|
||||
For each `etcd` host:
|
||||
|
||||
```bash
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
sudo kubeadm init phase certs etcd-server --config=/tmp/${HOST}/kubeadmcfg.yaml
|
||||
sudo kubeadm init phase certs etcd-peer --config=/tmp/${HOST}/kubeadmcfg.yaml
|
||||
sudo kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST}/kubeadmcfg.yaml
|
||||
sudo cp -R /etc/kubernetes/pki /tmp/${HOST}/
|
||||
sudo find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
|
||||
done
|
||||
```
|
||||
|
||||
### Startup the cluster
|
||||
Upload certificates on each `etcd` node and restart the `kubelet`
|
||||
|
||||
```bash
|
||||
for i in "${!ETCDHOSTS[@]}"; do
|
||||
HOST=${ETCDHOSTS[$i]}
|
||||
sudo chown -R ${USER}. /tmp/${HOST}
|
||||
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
|
||||
ssh ${USER}@${HOST} -t 'sudo chown -R root:root pki'
|
||||
ssh ${USER}@${HOST} -t 'sudo mv pki /etc/kubernetes/'
|
||||
ssh ${USER}@${HOST} -t 'sudo kubeadm init phase etcd local --config=kubeadmcfg.yaml'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl daemon-reload'
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl restart kubelet'
|
||||
done
|
||||
```
|
||||
|
||||
This will start the static `etcd` pod on each node and then the cluster gets formed.
|
||||
|
||||
Generate certificates for the `root` user
|
||||
|
||||
```bash
|
||||
cat > root-csr.json <<EOF
|
||||
{
|
||||
"CN": "root",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
```bash
|
||||
cfssl gencert \
|
||||
-ca=kamaji/etcd/ca.crt \
|
||||
-ca-key=kamaji/etcd/ca.key \
|
||||
-config=cfssl-cert-config.json \
|
||||
-profile=client-authentication \
|
||||
root-csr.json | cfssljson -bare root
|
||||
```
|
||||
|
||||
```bash
|
||||
cp root.pem kamaji/etcd/root.crt
|
||||
cp root-key.pem kamaji/etcd/root.key
|
||||
rm root*
|
||||
```
|
||||
|
||||
The result should be:
|
||||
|
||||
```bash
|
||||
$ tree kamaji
|
||||
kamaji
|
||||
└── etcd
|
||||
├── ca.crt
|
||||
├── ca.key
|
||||
├── root.crt
|
||||
└── root.key
|
||||
```
|
||||
|
||||
Use the `root` user to check the just formed `etcd` cluster is in health state
|
||||
|
||||
```bash
|
||||
export ETCDCTL_CACERT=kamaji/etcd/ca.crt
|
||||
export ETCDCTL_CERT=kamaji/etcd/root.crt
|
||||
export ETCDCTL_KEY=kamaji/etcd/root.key
|
||||
export ETCDCTL_ENDPOINTS=https://${ETCD0}:2379
|
||||
|
||||
etcdctl member list -w table
|
||||
```
|
||||
|
||||
The result should be something like this:
|
||||
|
||||
```
|
||||
+------------------+---------+--------+----------------------------+----------------------------+------------+
|
||||
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
|
||||
+------------------+---------+--------+----------------------------+----------------------------+------------+
|
||||
| 72657d6307364226 | started | etcd01 | https://192.168.32.11:2380 | https://192.168.32.11:2379 | false |
|
||||
| 91eb892c5ee87610 | started | etcd00 | https://192.168.32.10:2380 | https://192.168.32.10:2379 | false |
|
||||
| e9971c576949c34e | started | etcd02 | https://192.168.32.12:2380 | https://192.168.32.12:2379 | false |
|
||||
+------------------+---------+--------+----------------------------+----------------------------+------------+
|
||||
```
|
||||
|
||||
### Enable multi-tenancy
|
||||
The `root` user has full access to `etcd`, must be created before activating authentication. The `root` user must have the `root` role and is allowed to change anything inside `etcd`.
|
||||
|
||||
```bash
|
||||
etcdctl user add --no-password=true root
|
||||
etcdctl role add root
|
||||
etcdctl user grant-role root root
|
||||
etcdctl auth enable
|
||||
```
|
||||
|
||||
### Cleanup
|
||||
If you want to get rid of the etcd cluster, for each node, login and clean it:
|
||||
|
||||
```bash
|
||||
HOSTS=(${ETCD0} ${ETCD1} ${ETCD2})
|
||||
for i in "${!HOSTS[@]}"; do
|
||||
HOST=${HOSTS[$i]}
|
||||
ssh ${USER}@${HOST} -t 'sudo kubeadm reset -f';
|
||||
ssh ${USER}@${HOST} -t 'sudo systemctl reboot';
|
||||
done
|
||||
```
|
||||
|
||||
## Setup internal multi-tenant etcd
|
||||
If you opted for an internal etcd cluster running in the Kamaji admin cluster, follow steps below.
|
||||
|
||||
From the bootstrap machine load the environment for internal `etcd` setup:
|
||||
|
||||
```bash
|
||||
source kamaji-internal-etcd.env
|
||||
```
|
||||
|
||||
### Generate certificates
|
||||
On the bootstrap machine, using `kubeadm` init phase, create the `etcd` CA certificates:
|
||||
|
||||
```bash
|
||||
sudo kubeadm init phase certs etcd-ca
|
||||
mkdir kamaji
|
||||
sudo cp -r /etc/kubernetes/pki/etcd kamaji
|
||||
sudo chown -R ${USER}. kamaji/etcd
|
||||
```
|
||||
|
||||
Generate the `etcd` certificates for peers:
|
||||
|
||||
```
|
||||
cat << EOF | tee kamaji/etcd/peer-csr.json
|
||||
{
|
||||
"CN": "etcd",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
},
|
||||
"hosts": [
|
||||
"127.0.0.1",
|
||||
"etcd-0",
|
||||
"etcd-0.etcd",
|
||||
"etcd-0.etcd.${ETCD_NAMESPACE}.svc",
|
||||
"etcd-0.etcd.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-1",
|
||||
"etcd-1.etcd",
|
||||
"etcd-1.etcd.${ETCD_NAMESPACE}.svc",
|
||||
"etcd-1.etcd.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-2",
|
||||
"etcd-2.etcd",
|
||||
"etcd-2.etcd.${ETCD_NAMESPACE}.svc",
|
||||
"etcd-2.etcd.${ETCD_NAMESPACE}.cluster.local"
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
cfssl gencert -ca=kamaji/etcd/ca.crt -ca-key=kamaji/etcd/ca.key \
|
||||
-config=cfssl-cert-config.json \
|
||||
-profile=peer-authentication kamaji/etcd/peer-csr.json | cfssljson -bare kamaji/etcd/peer
|
||||
|
||||
```
|
||||
|
||||
Generate the `etcd` certificates for server:
|
||||
|
||||
```
|
||||
cat << EOF | tee kamaji/etcd/server-csr.json
|
||||
{
|
||||
"CN": "etcd",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
},
|
||||
"hosts": [
|
||||
"127.0.0.1",
|
||||
"etcd-server",
|
||||
"etcd-server.${ETCD_NAMESPACE}.svc",
|
||||
"etcd-server.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-0.etcd.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-1.etcd.${ETCD_NAMESPACE}.svc.cluster.local",
|
||||
"etcd-2.etcd.${ETCD_NAMESPACE}.svc.cluster.local"
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
cfssl gencert -ca=kamaji/etcd/ca.crt -ca-key=kamaji/etcd/ca.key \
|
||||
-config=cfssl-cert-config.json \
|
||||
-profile=peer-authentication kamaji/etcd/server-csr.json | cfssljson -bare kamaji/etcd/server
|
||||
```
|
||||
|
||||
Generate certificates for the `root` user of the `etcd`
|
||||
|
||||
```
|
||||
cat << EOF | tee kamaji/etcd/root-csr.json
|
||||
{
|
||||
"CN": "root",
|
||||
"key": {
|
||||
"algo": "rsa",
|
||||
"size": 2048
|
||||
}
|
||||
}
|
||||
EOF
|
||||
|
||||
cfssl gencert -ca=kamaji/etcd/ca.crt -ca-key=kamaji/etcd/ca.key \
|
||||
-config=cfssl-cert-config.json \
|
||||
-profile=client-authentication kamaji/etcd/root-csr.json | cfssljson -bare kamaji/etcd/root
|
||||
```
|
||||
|
||||
Install the `etcd` in the Kamaji admin cluster
|
||||
|
||||
```bash
|
||||
kubectl create namespace ${ETCD_NAMESPACE}
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} create secret generic etcd-certs \
|
||||
--from-file=kamaji/etcd/ca.crt \
|
||||
--from-file=kamaji/etcd/ca.key \
|
||||
--from-file=kamaji/etcd/peer-key.pem --from-file=kamaji/etcd/peer.pem \
|
||||
--from-file=kamaji/etcd/server-key.pem --from-file=kamaji/etcd/server.pem
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} apply -f etcd/etcd-cluster.yaml
|
||||
```
|
||||
|
||||
Install an `etcd` client to interact with the `etcd` server
|
||||
|
||||
```bash
|
||||
kubectl -n ${ETCD_NAMESPACE} create secret tls root-certs \
|
||||
--key=kamaji/etcd/root-key.pem \
|
||||
--cert=kamaji/etcd/root.pem
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} apply -f etcd/etcd-client.yaml
|
||||
```
|
||||
|
||||
Wait the etcd instances discover each other and the cluster is formed:
|
||||
|
||||
```bash
|
||||
kubectl -n ${ETCD_NAMESPACE} wait pod --for=condition=ready -l app=etcd --timeout=120s
|
||||
echo -n "\nChecking endpoint's health..."
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- /bin/bash -c "etcdctl endpoint health 1>/dev/null 2>/dev/null; until [ \$$? -eq 0 ]; do sleep 10; printf "."; etcdctl endpoint health 1>/dev/null 2>/dev/null; done;"
|
||||
echo -n "\netcd cluster's health:\n"
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- /bin/bash -c "etcdctl endpoint health"
|
||||
echo -n "\nWaiting for all members..."
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- /bin/bash -c "until [ \$$(etcdctl member list 2>/dev/null | wc -l) -eq 3 ]; do sleep 10; printf '.'; done;"
|
||||
@echo -n "\netcd's members:\n"
|
||||
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- /bin/bash -c "etcdctl member list -w table"
|
||||
```
|
||||
|
||||
### Enable multi-tenancy
|
||||
The `root` user has full access to `etcd`, must be created before activating authentication. The `root` user must have the `root` role and is allowed to change anything inside `etcd`.
|
||||
|
||||
```bash
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- etcdctl user add --no-password=true root
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- etcdctl role add root
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- etcdctl user grant-role root root
|
||||
kubectl -n ${ETCD_NAMESPACE} exec etcd-root-client -- etcdctl auth enable
|
||||
```
|
||||
|
||||
|
||||
## Install Kamaji controller
|
||||
Currently, the behaviour of the Kamaji controller for Tenant Control Plane is controlled by (in this order):
|
||||
|
||||
- CLI flags
|
||||
- Environment variables
|
||||
- Configuration file `kamaji.yaml` built into the image
|
||||
|
||||
By default Kamaji search for the configuration file and uses parameters found inside of it. In case some environment variable are passed, this will override configuration file parameters. In the end, if also a CLI flag is passed, this will override both env vars and config file as well.
|
||||
|
||||
There are multiple ways to deploy the Kamaji controller:
|
||||
|
||||
- Use the single YAML file installer
|
||||
- Use Kustomize with Makefile
|
||||
- Use the Kamaji Helm Chart
|
||||
|
||||
The Kamaji controller needs to access the multi-tenant `etcd` in order to provision the access for tenant `kube-apiserver`.
|
||||
|
||||
Create the secrets containing the `etcd` certificates
|
||||
|
||||
```bash
|
||||
kubectl create namespace kamaji-system
|
||||
kubectl -n kamaji-system create secret generic etcd-certs \
|
||||
--from-file=kamaji/etcd/ca.crt \
|
||||
--from-file=kamaji/etcd/ca.key
|
||||
|
||||
kubectl -n kamaji-system create secret tls root-client-certs \
|
||||
--cert=kamaji/etcd/root.crt \
|
||||
--key=kamaji/etcd/root.key
|
||||
```
|
||||
|
||||
### Install with a single manifest
|
||||
Install with the single YAML file installer:
|
||||
|
||||
```bash
|
||||
kubectl -n kamaji-system apply -f ../config/install.yaml
|
||||
```
|
||||
|
||||
Make sure to patch the `etcd` endpoints of the Kamaji controller, according to your environment:
|
||||
|
||||
```bash
|
||||
cat > patch-deploy.yaml <<EOF
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: manager
|
||||
args:
|
||||
- --health-probe-bind-address=:8081
|
||||
- --metrics-bind-address=127.0.0.1:8080
|
||||
- --leader-elect
|
||||
- --etcd-endpoints=${ETCD0}:2379,${ETCD1}:2379,${ETCD2}:2379
|
||||
EOF
|
||||
|
||||
kubectl -n kamaji-system patch \
|
||||
deployment kamaji-controller-manager \
|
||||
--patch-file patch-deploy.yaml
|
||||
```
|
||||
|
||||
The Kamaji Tenant Control Plane controller is now running on the Admin Cluster:
|
||||
|
||||
```bash
|
||||
kubectl -n kamaji-system get deploy
|
||||
NAME READY UP-TO-DATE AVAILABLE AGE
|
||||
operator-controller-manager 1/1 1 1 14h
|
||||
```
|
||||
|
||||
## Setup Tenant Cluster
|
||||
Now you are getting an Admin Cluster available to run multiple Tenant Control Planes, deployed by the Kamaji controller. Please, refer to the Kamaji Tenant Deployment [guide](./kamaji-tenant-deployment-guide.md).
|
||||
|
||||
@@ -67,6 +67,11 @@ spec:
|
||||
podCidr: ${TENANT_POD_CIDR}
|
||||
dnsServiceIPs:
|
||||
- ${TENANT_DNS_SERVICE}
|
||||
addons:
|
||||
coreDNS:
|
||||
enabled: true
|
||||
kubeProxy:
|
||||
enabled: true
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user