feat(traefik): Working PoC using single node Kubernetes cluster with Cilium, MetalLB and Traefik

QUICKSTART.md

@@ -0,0 +1,76 @@
+# Kubernetes
+
+## Disable swap
+
+```shell
+swapoff -a
+```
+
+## Start Kubernetes
+
+```shell
+sudo kubeadm init
+```
+
+## Set up kubectl
+
+```shell
+sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config
+```
+
+## Remove taint for single node use
+
+```shell
+kubectl taint nodes --all node-role.kubernetes.io/control-plane-
+```
+
+# Cilium 
+
+## Install Cilium as a CNI
+
+```shell
+cilium install
+```
+
+# Load Balancer
+
+## Install MetalLB for LoadBalancing
+https://raw.githubusercontent.com/metallb/metallb/v0.13.5/config/manifests/metallb-native.yaml
+```shell
+kubectl apply -f metallb/00-manifest.yml
+```
+
+## Configure MetalLB
+
+```shell
+kubectl apply -f metallb/02-configration.yml
+```
+
+# Traefik
+
+https://doc.traefik.io/traefik/v2.8/user-guides/crd-acme/
+
+## Create Traefik CRDs
+
+```shell
+kubectl apply -f traefik/00-crd-definition.yml
+kubectl apply -f traefik/01-crd-rbac.yml
+```
+
+## Create Service
+
+```shell
+kubectl apply -f traefik/02-service.yml
+```
+
+## Create Deployment
+
+```shell
+kubectl apply -f traefik/03-deployment.yml
+```
+
+## Create test application "whoami" with IngressRoutes
+
+```shell
+kubectl apply -f whoami/00-whoami.yml
+```

README.md

@@ -1,24 +1,13 @@
-A Terraform script to provision a Kubernetes Cluster with stuff
-
-
-# MAYBE JUST USE MINIKUBE?
-
-```
-minikube start --network-plugin=cni --cni=false
-```
-
-Need CNI (Cilium) LoadBalancer (MetaLB) and IngressController (Traefik) I think.
-https://pgillich.medium.com/setup-on-premise-kubernetes-with-kubeadm-metallb-traefik-and-vagrant-8a9d8d28951a
-
-Interesting: https://github.com/Mosibi/mosibi-kubernetes
-
 # Setup cluster with kubeadm
 
 Disable swap for kubelet to work properly
+
 ```shell
 swapoff -a
 ```
 
+## Install prerequisites
+
 ```shell
 sudo apt-get update
 sudo apt-get install -y apt-transport-https ca-certificates curl
@@ -32,16 +21,18 @@ sudo apt-get install -y containerd conntrack socat kubelet kubeadm kubectl
 ```
 
 cri-ctl: https://github.com/kubernetes-sigs/cri-tools
-nerdctl?
+TODO: nerdctl?
-
 
 We are going to use Cilium kube-proxy (TODO)
+
+## Initialise cluster
+
 ```shell
-sudo kubeadm init --skip-phases=addon/kube-proxy (TODO)
 sudo kubeadm init 
 ```
 
 ## Set up kubectl
+
 https://kubernetes.io/docs/tasks/tools/
 
 ```shell
@@ -49,61 +40,147 @@ mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config
 ```
 
-For remote kubectl
+For remote kubectl copy the config file to local machine
+
 ```shell
 scp veh@192.168.1.12:/home/veh/.kube/config ~/.kube/config
 ```
 
 ## (Optional) Remove taint for single node use
+
+Get taints on nodes
+
 ```shell
-kubectl taint nodes --all node-role.kubernetes.io/control-plane- node-role.kubernetes.io/master-
+kubectl get nodes -o json | jq '.items[].spec.taints'
 ```
 
-## Install CNI
+Remove taint on master node to allow scheduling of all deployments
-We choose Cilium
+
-https://docs.cilium.io/en/stable/gettingstarted/k8s-install-helm/
+```shell
+kubectl taint nodes --all node-role.kubernetes.io/control-plane-
+```
+
+## Install Cilium as Container Network Interface (CNI)
+
+https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/
+
+Install Cilium CLI
+
+```shell
+CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
+CLI_ARCH=amd64
+if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
+curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
+sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
+sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
+rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
+```
+
+Install Cilium
 
 ```shell
 cilium install
 ```
 
-```shell
+Validate install
-helm repo add cilium https://helm.cilium.io/
-```
 
 ```shell
-kubectl -n kube-system get pods --watch
+cilium status
 ```
 
-### Validate
+### (Optional) Replace kube-proxy with Cilium [TODO]
+https://docs.cilium.io/en/v1.12/gettingstarted/kubeproxy-free/
+
+*NB* Cluster should be initialised with 
+
 ```shell
-kubectl -n kube-system get pods -l k8s-app=cilium
+sudo kubeadm init --skip-phases=addon/kube-proxy
 ```
 
+
 ## MetalLB
+
+For load balancing
+
+https://metallb.universe.tf/installation/
+
+Installation
+https://raw.githubusercontent.com/metallb/metallb/v0.13.5/config/manifests/metallb-native.yaml
+
 ```shell
-helm repo add metallb https://metallb.github.io/metallb
+kubectl apply -f metallb/00-manifest.yml
-helm install metallb metallb/metallb
 ```
 
-## Deploy using Terraform
+Configure IP-pool and advertise as Level 2
-https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started
+https://metallb.universe.tf/configuration/
-```shell
+
-terraform plan 
+```yaml
-terraform apply
+kubectl apply -f metallb/02-configuration
 ```
 
-## Traefik IngressRoute CRD
+# Traefik
-https://doc.traefik.io/traefik/v2.0/routing/providers/kubernetes-crd/
-```shell
 
+## Traefik IngressRoute Custom Resource Definition (CRD)
+
+https://doc.traefik.io/traefik/v2.8/routing/providers/kubernetes-crd/
+https://doc.traefik.io/traefik/v2.8/user-guides/crd-acme/
+
+Create Custom Resource Definitions for Traefik
+
+```shell
+kubectl apply -f traefik/00-crd-definition.yml
+kubectl apply -f traefik/01-crd-rbac.yml
 ```
 
+## Service
+
+Create service for exposing Traefik deployment
+
+```shell
+kubectl apply -f traefik/02-service.yml
+```
+
+## Deployment
+
+Create deployment for Traefik
+
+```shell
+kubectl apply -f traefik/03-deployment.yml
+```
+
+## Port forward Traefik
+
+Port forward Traefik ports from 8000 to 80 for http and 4443 to 443 for https.
+IP can be found with `kubectl get svc`.
+
+# Test-application
+
+Create a test-application with
+
+```shell
+kubectl apply -f whoami/00-whoami.yml
+```
+
+`whoami` should now be available using https at `https://test.ratatoskr.myddns.rocks/tls`
+and using http at `http://test.ratatoskr.myddns.rocks/notls`.
+
+# Cleanup
 
-## Cleanup
 ```shell
 kubectl drain ratatoskr --delete-emptydir-data --force --ignore-daemonsets
 sudo kubeadm reset
 sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
 sudo ipvsadm -C
 ```
+
+# TODO
+
+## Deploy using Terraform
+
+https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started
+
+```shell
+terraform plan 
+terraform apply
+```
+

RESOURCES.md

@@ -1,4 +1,6 @@
-
 https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
+https://metallb.universe.tf/installation/
+https://kubernetes.io/docs/concepts/services-networking/service/
+https://doc.traefik.io/traefik/v2.8/user-guides/crd-acme/
 
 https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started

crd.yaml

@@ -1,34 +0,0 @@
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  name: ingressroutes.traefik.containo.us
-
-spec:
-  group: traefik.containo.us
-  versions:
-    - name: v1
-      served: true
-      storage: true
-  names:
-    kind: IngressRoute
-    plural: ingressroutes
-    singular: ingressroute
-  scope: Namespaced
-
----
-apiVersion: apiextensions.k8s.io/v1
-kind: CustomResourceDefinition
-metadata:
-  name: ingressroutetcps.traefik.containo.us
-
-spec:
-  group: traefik.containo.us
-  versions:
-    - name: v1
-      served: true
-      storage: true
-  names:
-    kind: IngressRouteTCP
-    plural: ingressroutetcps
-    singular: ingressroutetcp
-  scope: Namespaced

dashboard.yaml

@@ -1,14 +0,0 @@
-# dashboard.yaml
-apiVersion: traefik.containo.us/v1alpha1
-kind: IngressRoute
-metadata:
-  name: dashboard
-spec:
-  entryPoints:
-    - web
-  routes:
-    - match: Host(`traefik.ratatoskr.myddns.rocks`)
-      kind: Rule
-      services:
-        - name: api@internal
-          kind: TraefikService

helm/metallb-values.yaml

@@ -1,6 +0,0 @@
-configInline:
-  address-pools:
-    - name: default
-      protocol: layer2
-      addresses:
-        - 192.168.1.240-192.168.1.250

main.tf

@@ -30,41 +30,41 @@ provider "helm" {
 #  version    = "1.11.5"
 #}
 
-resource "kubernetes_namespace" "traefik" {
+#resource "kubernetes_namespace" "traefik" {
-  metadata {
+#  metadata {
-    name = "traefik"
+#    name = "traefik"
-  }
+#  }
-}
+#}
-
+#
-resource "helm_release" "traefik" {
+#resource "helm_release" "traefik" {
-  name = "traefik"
+#  name = "traefik"
-
+#
-  repository = "https://helm.traefik.io/traefik"
+#  repository = "https://helm.traefik.io/traefik"
-  chart      = "traefik"
+#  chart      = "traefik"
-  namespace  = "traefik"
+#  namespace  = "traefik"
-  version    = "10.20.0"
+#  version    = "10.20.0"
-
+#
-}
+#}
-
+#
-resource "kubernetes_service" "traefik" {
+#resource "kubernetes_service" "traefik" {
-  metadata {
+#  metadata {
-    name      = "traefik"
+#    name      = "traefik"
-    namespace = kubernetes_namespace.traefik.metadata.0.name
+#    namespace = kubernetes_namespace.traefik.metadata.0.name
-  }
+#  }
-  spec {
+#  spec {
-    selector = {
+#    selector = {
-      # Standard Helm chart label to locate pods
+#      # Standard Helm chart label to locate pods
-      "app.kubernetes.io/name" = helm_release.traefik.name
+#      "app.kubernetes.io/name" = helm_release.traefik.name
-    }
+#    }
-
+#
-    type = "LoadBalancer"
+#    type = "LoadBalancer"
-    port {
+#    port {
-      protocol    = "TCP"
+#      protocol    = "TCP"
-      port        = 80    # External exposed port to reach container
+#      port        = 80    # External exposed port to reach container
-      target_port = 9000  # Internal exposed port of container
+#      target_port = 9000  # Internal exposed port of container
-    }
+#    }
-  }
+#  }
-}
+#}
 
 resource "kubernetes_namespace" "test" {
   metadata {

metallb/01-configuration.yml

@@ -0,0 +1,17 @@
+# Configure IP-pool
+apiVersion: metallb.io/v1beta1
+kind: IPAddressPool
+metadata:
+  name: first-pool
+  namespace: metallb-system
+spec:
+  addresses:
+    - 192.168.1.240-192.168.1.250
+
+---
+# Advertise
+apiVersion: metallb.io/v1beta1
+kind: L2Advertisement
+metadata:
+  name: example
+  namespace: metallb-system

traefik/01-crd-rbac.yml

@@ -0,0 +1,65 @@
+# https://raw.githubusercontent.com/traefik/traefik/v2.8/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: traefik-ingress-controller
+
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - services
+      - endpoints
+      - secrets
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - extensions
+      - networking.k8s.io
+    resources:
+      - ingresses
+      - ingressclasses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - extensions
+      - networking.k8s.io
+    resources:
+      - ingresses/status
+    verbs:
+      - update
+  - apiGroups:
+      - traefik.containo.us
+    resources:
+      - middlewares
+      - middlewaretcps
+      - ingressroutes
+      - traefikservices
+      - ingressroutetcps
+      - ingressrouteudps
+      - tlsoptions
+      - tlsstores
+      - serverstransports
+    verbs:
+      - get
+      - list
+      - watch
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: traefik-ingress-controller
+
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: traefik-ingress-controller
+subjects:
+  - kind: ServiceAccount
+    name: traefik-ingress-controller
+    namespace: default

traefik/02-service.yml

@@ -0,0 +1,19 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: traefik
+
+spec:
+  type: LoadBalancer
+  ports:
+    - protocol: TCP
+      name: web
+      port: 8000
+    - protocol: TCP
+      name: admin
+      port: 8080
+    - protocol: TCP
+      name: websecure
+      port: 4443
+  selector:
+    app: traefik

traefik/03-deployment.yml

@@ -0,0 +1,48 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  namespace: default
+  name: traefik-ingress-controller
+
+---
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  namespace: default
+  name: traefik
+  labels:
+    app: traefik
+
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: traefik
+  template:
+    metadata:
+      labels:
+        app: traefik
+    spec:
+      serviceAccountName: traefik-ingress-controller
+      containers:
+        - name: traefik
+          image: traefik:v2.8
+          args:
+            - --api.insecure
+            - --accesslog
+            - --entrypoints.web.Address=:8000
+            - --entrypoints.websecure.Address=:4443
+            - --providers.kubernetescrd
+            - --certificatesresolvers.myresolver.acme.tlschallenge
+            - --certificatesresolvers.myresolver.acme.email=veghag@gmail.com
+            - --certificatesresolvers.myresolver.acme.storage=acme.json
+            # Please note that this is the staging Let's Encrypt server.
+            # Once you get things working, you should remove that whole line altogether.
+            #- --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
+          ports:
+            - name: web
+              containerPort: 8000
+            - name: websecure
+              containerPort: 4443
+            - name: admin
+              containerPort: 8080

whoami/00-whoami.yml

@@ -0,0 +1,78 @@
+---
+# Service for exposing deployment of whoami
+apiVersion: v1
+kind: Service
+metadata:
+  name: whoami
+
+spec:
+  type: LoadBalancer
+  ports:
+    - protocol: TCP
+      name: web
+      port: 80
+  selector:
+    app: whoami
+
+---
+# Deployment of whoami
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  namespace: default
+  name: whoami
+  labels:
+    app: whoami
+
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: whoami
+  template:
+    metadata:
+      labels:
+        app: whoami
+    spec:
+      containers:
+        - name: whoami
+          image: traefik/whoami
+          ports:
+            - name: web
+              containerPort: 80
+
+---
+# IngressRoute for insecure whoami address
+apiVersion: traefik.containo.us/v1alpha1
+kind: IngressRoute
+metadata:
+  name: simpleingressroute
+  namespace: default
+spec:
+  entryPoints:
+    - web
+  routes:
+    - match: Host(`test.ratatoskr.myddns.rocks`) && PathPrefix(`/notls`)
+      kind: Rule
+      services:
+        - name: whoami
+          port: 80
+
+---
+# IngressRoute for secure whoami address
+apiVersion: traefik.containo.us/v1alpha1
+kind: IngressRoute
+metadata:
+  name: ingressroutetls
+  namespace: default
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`test.ratatoskr.myddns.rocks`) && PathPrefix(`/tls`)
+      kind: Rule
+      services:
+        - name: whoami
+          port: 80
+  tls:
+    certResolver: myresolver