scottk/homelab
1
0
Fork 0
mirror of https://github.com/optim-enterprises-bv/homelab.git synced 2025-10-29 09:02:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(traefik): Working PoC using single node Kubernetes cluster with Cilium, MetalLB and Traefik

Browse Source
This commit is contained in:
Vegard Hagen
2022-09-18 20:30:41 +02:00
parent 5a716387a8
commit 482f35c139
14 changed files with 4523 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files

76
QUICKSTART.md Normal file
View File

@@ -0,0 +1,76 @@
# Kubernetes
## Disable swap
```shell
swapoff -a
```
## Start Kubernetes
```shell
sudo kubeadm init
```
## Set up kubectl
```shell
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
## Remove taint for single node use
```shell
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
```
# Cilium
## Install Cilium as a CNI
```shell
cilium install
```
# Load Balancer
## Install MetalLB for LoadBalancing
https://raw.githubusercontent.com/metallb/metallb/v0.13.5/config/manifests/metallb-native.yaml
```shell
kubectl apply -f metallb/00-manifest.yml
```
## Configure MetalLB
```shell
kubectl apply -f metallb/02-configration.yml
```
# Traefik
https://doc.traefik.io/traefik/v2.8/user-guides/crd-acme/
## Create Traefik CRDs
```shell
kubectl apply -f traefik/00-crd-definition.yml
kubectl apply -f traefik/01-crd-rbac.yml
```
## Create Service
```shell
kubectl apply -f traefik/02-service.yml
```
## Create Deployment
```shell
kubectl apply -f traefik/03-deployment.yml
```
## Create test application "whoami" with IngressRoutes
```shell
kubectl apply -f whoami/00-whoami.yml
```

157
README.md
View File

@@ -1,24 +1,13 @@
A Terraform script to provision a Kubernetes Cluster with stuff
# MAYBE JUST USE MINIKUBE?
```
minikube start --network-plugin=cni --cni=false
```
Need CNI (Cilium) LoadBalancer (MetaLB) and IngressController (Traefik) I think.
https://pgillich.medium.com/setup-on-premise-kubernetes-with-kubeadm-metallb-traefik-and-vagrant-8a9d8d28951a
Interesting: https://github.com/Mosibi/mosibi-kubernetes
# Setup cluster with kubeadm
Disable swap for kubelet to work properly
```shell
swapoff -a
```
## Install prerequisites
```shell
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
@@ -32,16 +21,18 @@ sudo apt-get install -y containerd conntrack socat kubelet kubeadm kubectl
```
cri-ctl: https://github.com/kubernetes-sigs/cri-tools
nerdctl?
TODO: nerdctl?
We are going to use Cilium kube-proxy (TODO)
## Initialise cluster
```shell
sudo kubeadm init --skip-phases=addon/kube-proxy (TODO)
sudo kubeadm init
```
## Set up kubectl
https://kubernetes.io/docs/tasks/tools/
```shell
@@ -49,61 +40,147 @@ mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
For remote kubectl
For remote kubectl copy the config file to local machine
```shell
scp veh@192.168.1.12:/home/veh/.kube/config ~/.kube/config
```
## (Optional) Remove taint for single node use
Get taints on nodes
```shell
kubectl taint nodes --all node-role.kubernetes.io/control-plane- node-role.kubernetes.io/master-
kubectl get nodes -o json | jq '.items[].spec.taints'
```
## Install CNI
We choose Cilium
https://docs.cilium.io/en/stable/gettingstarted/k8s-install-helm/
Remove taint on master node to allow scheduling of all deployments
```shell
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
```
## Install Cilium as Container Network Interface (CNI)
https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/
Install Cilium CLI
```shell
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
```
Install Cilium
```shell
cilium install
```
```shell
helm repo add cilium https://helm.cilium.io/
```
Validate install
```shell
kubectl -n kube-system get pods --watch
cilium status
```
### Validate
### (Optional) Replace kube-proxy with Cilium [TODO]
https://docs.cilium.io/en/v1.12/gettingstarted/kubeproxy-free/
*NB* Cluster should be initialised with
```shell
kubectl -n kube-system get pods -l k8s-app=cilium
sudo kubeadm init --skip-phases=addon/kube-proxy
```
## MetalLB
For load balancing
https://metallb.universe.tf/installation/
Installation
https://raw.githubusercontent.com/metallb/metallb/v0.13.5/config/manifests/metallb-native.yaml
```shell
helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb
kubectl apply -f metallb/00-manifest.yml
```
## Deploy using Terraform
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started
```shell
terraform plan
terraform apply
Configure IP-pool and advertise as Level 2
https://metallb.universe.tf/configuration/
```yaml
kubectl apply -f metallb/02-configuration
```
## Traefik IngressRoute CRD
https://doc.traefik.io/traefik/v2.0/routing/providers/kubernetes-crd/
```shell
# Traefik
## Traefik IngressRoute Custom Resource Definition (CRD)
https://doc.traefik.io/traefik/v2.8/routing/providers/kubernetes-crd/
https://doc.traefik.io/traefik/v2.8/user-guides/crd-acme/
Create Custom Resource Definitions for Traefik
```shell
kubectl apply -f traefik/00-crd-definition.yml
kubectl apply -f traefik/01-crd-rbac.yml
```
## Service
Create service for exposing Traefik deployment
```shell
kubectl apply -f traefik/02-service.yml
```
## Deployment
Create deployment for Traefik
```shell
kubectl apply -f traefik/03-deployment.yml
```
## Port forward Traefik
Port forward Traefik ports from 8000 to 80 for http and 4443 to 443 for https.
IP can be found with `kubectl get svc`.
# Test-application
Create a test-application with
```shell
kubectl apply -f whoami/00-whoami.yml
```
`whoami` should now be available using https at `https://test.ratatoskr.myddns.rocks/tls`
and using http at `http://test.ratatoskr.myddns.rocks/notls`.
# Cleanup
## Cleanup
```shell
kubectl drain ratatoskr --delete-emptydir-data --force --ignore-daemonsets
sudo kubeadm reset
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
sudo ipvsadm -C
```
```
# TODO
## Deploy using Terraform
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started
```shell
terraform plan
terraform apply
```

4
RESOURCES.md
View File

@@ -1,4 +1,6 @@
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
https://metallb.universe.tf/installation/
https://kubernetes.io/docs/concepts/services-networking/service/
https://doc.traefik.io/traefik/v2.8/user-guides/crd-acme/
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started

34
crd.yaml
View File

@@ -1,34 +0,0 @@
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: ingressroutes.traefik.containo.us
spec:
group: traefik.containo.us
versions:
- name: v1
served: true
storage: true
names:
kind: IngressRoute
plural: ingressroutes
singular: ingressroute
scope: Namespaced
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: ingressroutetcps.traefik.containo.us
spec:
group: traefik.containo.us
versions:
- name: v1
served: true
storage: true
names:
kind: IngressRouteTCP
plural: ingressroutetcps
singular: ingressroutetcp
scope: Namespaced

14
dashboard.yaml
View File

@@ -1,14 +0,0 @@
# dashboard.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: dashboard
spec:
entryPoints:
- web
routes:
- match: Host(`traefik.ratatoskr.myddns.rocks`)
kind: Rule
services:
- name: api@internal
kind: TraefikService

6
helm/metallb-values.yaml
View File

@@ -1,6 +0,0 @@
configInline:
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.1.240-192.168.1.250

70
main.tf
View File

@@ -30,41 +30,41 @@ provider "helm" {
# version = "1.11.5"
#}
resource "kubernetes_namespace" "traefik" {
metadata {
name = "traefik"
}
}
resource "helm_release" "traefik" {
name = "traefik"
repository = "https://helm.traefik.io/traefik"
chart = "traefik"
namespace = "traefik"
version = "10.20.0"
}
resource "kubernetes_service" "traefik" {
metadata {
name = "traefik"
namespace = kubernetes_namespace.traefik.metadata.0.name
}
spec {
selector = {
# Standard Helm chart label to locate pods
"app.kubernetes.io/name" = helm_release.traefik.name
}
type = "LoadBalancer"
port {
protocol = "TCP"
port = 80 # External exposed port to reach container
target_port = 9000 # Internal exposed port of container
}
}
}
#resource "kubernetes_namespace" "traefik" {
# metadata {
# name = "traefik"
# }
#}
#
#resource "helm_release" "traefik" {
# name = "traefik"
#
# repository = "https://helm.traefik.io/traefik"
# chart = "traefik"
# namespace = "traefik"
# version = "10.20.0"
#
#}
#
#resource "kubernetes_service" "traefik" {
# metadata {
# name = "traefik"
# namespace = kubernetes_namespace.traefik.metadata.0.name
# }
# spec {
# selector = {
# # Standard Helm chart label to locate pods
# "app.kubernetes.io/name" = helm_release.traefik.name
# }
#
# type = "LoadBalancer"
# port {
# protocol = "TCP"
# port = 80 # External exposed port to reach container
# target_port = 9000 # Internal exposed port of container
# }
# }
#}
resource "kubernetes_namespace" "test" {
metadata {

1799
metallb/00-manifest.yml Normal file
View File

File diff suppressed because it is too large Load Diff

17
metallb/01-configuration.yml Normal file
View File

@@ -0,0 +1,17 @@
# Configure IP-pool
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
namespace: metallb-system
spec:
addresses:
- 192.168.1.240-192.168.1.250
---
# Advertise
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: example
namespace: metallb-system

2266
traefik/00-crd-definition.yml Normal file
View File

File diff suppressed because it is too large Load Diff

65
traefik/01-crd-rbac.yml Normal file
View File

@@ -0,0 +1,65 @@
# https://raw.githubusercontent.com/traefik/traefik/v2.8/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
- middlewaretcps
- ingressroutes
- traefikservices
- ingressroutetcps
- ingressrouteudps
- tlsoptions
- tlsstores
- serverstransports
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: default

19
traefik/02-service.yml Normal file
View File

@@ -0,0 +1,19 @@
apiVersion: v1
kind: Service
metadata:
name: traefik
spec:
type: LoadBalancer
ports:
- protocol: TCP
name: web
port: 8000
- protocol: TCP
name: admin
port: 8080
- protocol: TCP
name: websecure
port: 4443
selector:
app: traefik

48
traefik/03-deployment.yml Normal file
View File

@@ -0,0 +1,48 @@
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: default
name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.8
args:
- --api.insecure
- --accesslog
- --entrypoints.web.Address=:8000
- --entrypoints.websecure.Address=:4443
- --providers.kubernetescrd
- --certificatesresolvers.myresolver.acme.tlschallenge
- --certificatesresolvers.myresolver.acme.email=veghag@gmail.com
- --certificatesresolvers.myresolver.acme.storage=acme.json
# Please note that this is the staging Let's Encrypt server.
# Once you get things working, you should remove that whole line altogether.
#- --certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
ports:
- name: web
containerPort: 8000
- name: websecure
containerPort: 4443
- name: admin
containerPort: 8080

78
whoami/00-whoami.yml Normal file
View File

@@ -0,0 +1,78 @@
---
# Service for exposing deployment of whoami
apiVersion: v1
kind: Service
metadata:
name: whoami
spec:
type: LoadBalancer
ports:
- protocol: TCP
name: web
port: 80
selector:
app: whoami
---
# Deployment of whoami
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: default
name: whoami
labels:
app: whoami
spec:
replicas: 2
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami
ports:
- name: web
containerPort: 80
---
# IngressRoute for insecure whoami address
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: simpleingressroute
namespace: default
spec:
entryPoints:
- web
routes:
- match: Host(`test.ratatoskr.myddns.rocks`) && PathPrefix(`/notls`)
kind: Rule
services:
- name: whoami
port: 80
---
# IngressRoute for secure whoami address
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: ingressroutetls
namespace: default
spec:
entryPoints:
- websecure
routes:
- match: Host(`test.ratatoskr.myddns.rocks`) && PathPrefix(`/tls`)
kind: Rule
services:
- name: whoami
port: 80
tls:
certResolver: myresolver