```W0330 12:41:41.898614 1 reflector.go:569] pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *endpoint.DNSEndpoint: dnsendpoints.externaldns.k8s.io is forbidden: User "
E0330 12:41:41.898748 1 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *endpoint.DNSEndpoint: failed to list *endpoint.DNS```
- On Intel Mac the homebrew binaries are located at /usr/local/bin/ instead of
/opt/homebrew/bin, let's include the Intel Mac path in the check as well.
- See https://docs.brew.sh/Installation for more details on possible install configurations.
- Checking the actual versions of these binaries would be better but this works fine for now.
* feat: only deploy spegel if more than one node
Signed-off-by: Devin Buhl <devin@buhl.casa>
* feat: only deploy spegel if more than one node
Signed-off-by: Devin Buhl <devin@buhl.casa>
---------
Signed-off-by: Devin Buhl <devin@buhl.casa>
* Fix: Allow secureboot and encrypt_disk fields to be optional
Updated the node schema to add optioanal secureboot and encrypt_disk fields in schema.
* Update nodes.schema.cue
---------
Co-authored-by: Devin Buhl <onedr0p@users.noreply.github.com>
In #1791 a `strip()` was added to a bunch of strings read from files. However, this also removes newlines, so for the github-deploy-key the `known_hosts` would end up on the same line as the end of the private key and therefore not work in the `source-controller`.
Instead of removing the `strip()` let's remove the `-` sign in the jinja end block so we keep newline after it.
Welcome to my opinionated and extensible template for deploying a single Kubernetes cluster. The goal of this project is to make it easier for people interested in using Kubernetes to deploy a cluster at home on bare-metal or VMs.
Welcome to my template designed for deploying a single Kubernetes cluster. Whether you're setting up a cluster at home on bare-metal or virtual machines (VMs), this project aims to simplify the process and make Kubernetes more accessible. This template is inspired by my personal [home-ops](https://github.com/onedr0p/home-ops) repository, providing a practical starting point for anyone interested in managing their own Kubernetes environment.
At a high level this project makes use of [makejinja](https://github.com/mirkolenz/makejinja) to read in a [configuration file](./config.sample.yaml) which renders out templates that will allow you to install and manage your Kubernetes cluster with.
At its core, this project leverages [makejinja](https://github.com/mirkolenz/makejinja), a powerful tool for rendering templates. By reading configuration files—such as [cluster.yaml](./cluster.sample.yaml) and [nodes.yaml](./nodes.sample.yaml)—Makejinja generates the necessary configurations to deploy a Kubernetes cluster with the following features:
- Easy configuration through YAML files.
- Compatibility with home setups, whether on physical hardware or VMs.
- A modular and extensible approach to cluster deployment and management.
With this approach, you'll gain a solid foundation to build and manage your Kubernetes cluster efficiently.
## ✨ Features
The features included will depend on the type of configuration you want to use. There are currently **2 different types** of **configurations** available with this template.
A Kubernetes cluster deployed with [Talos Linux](https://github.com/siderolabs/talos) and an opinionated implementation of [Flux](https://github.com/fluxcd/flux2) using [GitHub](https://github.com/) as the Git provider, [sops](https://github.com/getsops/sops) to manage secrets and [cloudflared](https://github.com/cloudflare/cloudflared) to access applications external to your local network.
1.**"Flux cluster"** - a Kubernetes cluster deployed on-top of [Talos Linux](https://github.com/siderolabs/talos) with an opinionated implementation of [Flux](https://github.com/fluxcd/flux2) using [GitHub](https://github.com/) as the Git provider and [sops](https://github.com/getsops/sops) to manage secrets.
- **Required:** Some knowledge of [Containers](https://opencontainers.org/), [YAML](https://yaml.org/), and [Git](https://git-scm.com/).
- **Components:** [flux](https://github.com/fluxcd/flux2), [Cilium](https://github.com/cilium/cilium),[cert-manager](https://github.com/cert-manager/cert-manager), [spegel](https://github.com/spegel-org/spegel), [reloader](https://github.com/stakater/Reloader), and [openebs](https://github.com/openebs/openebs).
2.**"Flux cluster with Cloudflare"** - An addition to "**Flux cluster**" that provides DNS and SSL with [Cloudflare](https://www.cloudflare.com/). [Cloudflare Tunnel](https://www.cloudflare.com/products/tunnel/) is also included to provide external access to certain applications deployed in your cluster.
- **Required:** A Cloudflare account with a domain managed in your Cloudflare account.
- **Components:** [ingress-nginx](https://github.com/kubernetes/ingress-nginx/), [external-dns](https://github.com/kubernetes-sigs/external-dns) and [cloudflared](https://github.com/cloudflare/cloudflared).
- **Required:** Some knowledge of [Containers](https://opencontainers.org/), [YAML](https://noyaml.com/), [Git](https://git-scm.com/), and a **Cloudflare account** with a **domain**.
-A [Renovate](https://www.mend.io/renovate)-ready repository with pull request diffs provided by [flux-local](https://github.com/allenporter/flux-local)
-Integrated [GitHub Actions](https://github.com/features/actions) with helpful workflows.
- Flux `HelmRelease` and `Kustomization` diffs w/ [flux-local](https://github.com/allenporter/flux-local)
## 💻 Machine Preparation
Does this sound cool to you? If so, continue to read on! 👇
### System requirements
## 🚀 Let's Go!
> [!NOTE]
> 1. The included behaviour of Talos is that all nodes are able to run workloads, **including** the controller nodes. **Worker nodes** are therefore **optional**.
> 2. Do you have 3 or more nodes? It is highly recommended to make 3 of them controller nodes for a highly available control plane.
> 3. Running the cluster on Proxmox VE? My thoughts and recommendations about that are documented [here](https://onedr0p.github.io/home-ops/notes/proxmox-considerations.html).
There are **6 stages** outlined below for completing this project, make sure you follow the stages in order.
1. Head over to <https://factory.talos.dev> and follow the instructions which will eventually lead you to download a Talos Linux iso file (or for SBCs the `.raw.xz`). Make sure to note the schematic ID you will need this later on.
For a **stable** and **high-availability** production Kubernetes cluster, hardware selection is critical. NVMe/SSDs are strongly preferred over HDDs, and **Bare Metal is strongly recommended** over virtualized platforms like Proxmox.
2. Flash the iso or raw file to a USB drive and boot to Talos on your nodes with it.
Using **enterprise NVMe or SATA SSDs on Bare Metal** (even used drives) provides the most reliable performance and rock-solid stability. Consumer **NVMe or SATA SSDs**, on the other hand, carry risks such as latency spikes, corruption, and fsync delays, particularly in multi-node setups.
3. Continue on to 🚀 [**Getting Started**](#-getting-started)
**Proxmox with enterprise drives can work** for testing or carefully tuned production clusters, but it introduces additional layers of potential I/O contention — especially if consumer drives are used. Any **replicated storage** (e.g., Rook-Ceph, Longhorn) should always use **dedicated disks separate from control plane and etcd nodes** to ensure reliability. Worker nodes are more flexible, but risky configurations should still be avoided for stateful workloads to maintain cluster stability.
## 🚀 Getting Started
These guidelines provide a strong baseline, but there are always exceptions and nuances. The best way to ensure your hardware configuration works is to **test it thoroughly and benchmark performance** under realistic workloads.
Once you have installed Talos on your nodes, there are six stages to getting a Flux-managed cluster up and runnning.
### Stage 2: Machine Preparation
> [!NOTE]
> For all stages below the commands **MUST** be ran on your personal workstation within your repository directory
> [!IMPORTANT]
> If you have **3 or more nodes** it is recommended to make 3 of them controller nodes for a highly available control plane. This project configures **all nodes** to be able to run workloads. **Worker nodes** are therefore **optional**.
1. Head over to the [Talos Linux Image Factory](https://factory.talos.dev) and follow the instructions. Be sure to only choose the **bare-minimum system extensions** as some might require additional configuration and prevent Talos from booting without it. Depending on your CPU start with the Intel/AMD system extensions (`i915`, `intel-ucode` & `mei`**or**`amdgpu` & `amd-ucode`), you can always add system extensions after Talos is installed and working.
1.Create a new **public** repository by clicking the big green "Use this template" button at the top of this page.
2.This will eventually lead you to download a Talos Linux ISO (or for SBCs a RAW) image. Make sure to note the **schematic ID** you will need this later on.
2.Clone **your new repo** to you local workstation and `cd` into it.
3.Flash the Talos ISO or RAW image to a USB drive and boot from it on your nodes.
3.Continue on to 🌱 [**Stage 2**](#-stage-2-setup-your-local-workstation-environment)
### 🌱 Stage 2: Setup your local workstation
You have two different options for setting up your local workstation.
- First option is using a `devcontainer` which requires you to have Docker and VSCode installed. This method is the fastest to get going because all the required CLI tools are provided for you in my [devcontainer](https://github.com/onedr0p/cluster-template/pkgs/container/cluster-template%2Fdevcontainer) image.
- The second option is setting up the CLI tools directly on your workstation.
#### Devcontainer method
1. Start Docker and open your repository in VSCode. There will be a pop-up asking you to use the `devcontainer`, click the button to start using it.
2. Continue on to 🔧 [**Stage 3**](#-stage-3-bootstrap-configuration)
#### Non-devcontainer method
1. Install the most recent version of [task](https://taskfile.dev/) and [direnv](https://direnv.net/)
4.Verify with `nmap` that your nodes are available on the network. (Replace `192.168.1.0/24` with the network your nodes are on.)
2. [Hook `direnv` into your preferred shell](https://direnv.net/docs/hook.html), then run:
### Stage 3: Local Workstation
> [!TIP]
> It is recommended to set the visibility of your repository to `Public` so you can easily request help if you get stuck.
1. Create a new repository by clicking the green `Use this template` button at the top of this page, then clone the new repo you just created and `cd` into it. Alternatively you can use the [GitHub CLI](https://cli.github.com/) ...
📍 _**Verify** that `direnv` is setup properly by opening a new terminal and `cd`ing into your repository. You should see something like:_
2. **Install** the [Mise CLI](https://mise.jdx.dev/getting-started.html#installing-mise-cli) on your local workstation.
3. **Activate** Mise in your shell by following the [activation guide](https://mise.jdx.dev/getting-started.html#activate-mise).
4. Use `mise` to install the **required** CLI tools:
```sh
cd /path/to/repo
direnv: loading ... .envrc
direnv: export +VIRTUAL_ENV ... ~PATH
mise trust
pip install pipx
mise install
```
3. Install the additional **required** CLI tools
📍 _**Having trouble installing the tools?** Try unsetting the `GITHUB_TOKEN` env var and then run these commands again_
📍 _**Not using Homebrew or ArchLinux?** Try using the generic Linux task below, if that fails check out the [Brewfile](.taskfiles/workstation/Brewfile)/[Archfile](.taskfiles/workstation/Archfile) for what CLI tools needed and install them._
📍 _**Having trouble compiling Python?** Try running `mise settings python.compile=0` and then run these commands again_
5. Logout of the GitHub Container Registry as this may cause authorization problems in future steps when using the public registry:
```sh
# Homebrew
task workstation:brew
# or, Arch with yay/paru
task workstation:arch
# or, Generic Linux (YMMV, this pulls binaires in to ./bin)
task workstation:generic-linux
docker logout ghcr.io
helm registry logout ghcr.io
```
4. Setup a Python virual environment by running the following task command.
### Stage 4: Cloudflare configuration
📍 _This commands requires Python 3.11+ to be installed._
> [!WARNING]
> If any of the commands fail with `command not found` or `unknown command` it means `mise` is either not installed, activated or it could be configured incorrectly.
1. Create a Cloudflare API token for use with cloudflared and external-dns by reviewing the official [documentation](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) and following the instructions below.
- Click the blue `Use template` button for the `Edit zone DNS` template.
- Name your token `kubernetes`
- Under `Permissions`, click `+ Add More` and add permissions `Zone - DNS - Edit` and `Account - Cloudflare Tunnel - Read`
- Limit the permissions to a specific account and/or zone resources and then click `Continue to Summary` and then `Create Token`.
- **Save this token somewhere safe**, you will need it later on.
5. Continue on to 🔧 [**Stage 3**](#-stage-3-bootstrap-configuration)
### Stage 5: Cluster configuration
### 🔧 Stage 3: Bootstrap configuration
> [!NOTE]
> The [config.sample.yaml](./config.sample.yaml) file contains config that is **vital** to the bootstrap process.
1. Generate the `config.yaml` from the [config.sample.yaml](./config.sample.yaml) configuration file.
1. Generate the config files from the sample files:
```sh
task init
```
2. Fill out the `config.yaml` configuration file using the comments in that file as a guide.
2. Fill out `cluster.yaml` and `nodes.yaml` configuration files using the comments in those file as a guide.
3. Run the following command which will generate all the files needed to continue.
3. Template out the kubernetes and talos configuration files, if any issues come up be sure to read the error and adjust your config files accordingly.
```sh
task configure
```
4. Push you changes to git
4. Push your changes to git:
📍 _**Verify** all the `./kubernetes/**/*.sops.*` files are **encrypted** with SOPS_
```sh
git add -A
git commit -m "Initial commit :rocket:"
git commit -m "chore: initial commit :rocket:"
git push
```
### ⛵ Stage 4: Install Kubernetes
> [!TIP]
> Using a **private repository**? Make sure to paste the public key from `github-deploy.key.pub` into the deploy keys section of your GitHub repository settings. This will make sure Flux has read/write access to your repository.
1. Deploy your cluster and bootstrap it. This generates secrets, generates the config files for your nodes and applies them. It bootstraps the cluster afterwards, fetches the kubeconfig file and installs Cilium and kubelet-csr-approver. It finishes with some health checks.
### Stage 6: Bootstrap Talos, Kubernetes, and Flux
> [!WARNING]
> It might take a while for the cluster to be setup (10+ minutes is normal). During which time you will see a variety of error messages like: "couldn't get current server API group list," "error: no matching resources found", etc. 'Ready' will remain "False" as no CNI is deployed yet. **This is normal.** If this step gets interrupted, e.g. by pressing <kbd>Ctrl</kbd> + <kbd>C</kbd>, you likely will need to [reset the cluster](#-reset) before trying again
1. Install Talos:
```sh
task bootstrap:talos
```
2. ⚠️ It might take a while for the cluster to be setup (10+ minutes is normal), during which time you will see a variety of error messages like: "couldn't get current server API group list," "error: no matching resources found", etc. This is a normal. If this step gets interrupted, e.g. by pressing <kbd>Ctrl</kbd> + <kbd>C</kbd>, you likely will need to [nuke the cluster](#-Nuke) before trying again.
#### Cluster validation
1. The `kubeconfig` for interacting with your cluster should have been created in the root of your repository.
2. Verify the nodes are online
📍 _If this command **fails** you likely haven't configured `direnv` as [mentioned previously](#non-devcontainer-method) in the guide._
2. Check the status of Flux and if the Flux resources are up-to-date and in a ready state:
_Mic check, 1, 2_ - In a few moments applications should be lighting up like Christmas in July 🎄
1. Output all the common resources in your cluster.
📍 _Feel free to use the provided [kubernetes tasks](.taskfiles/Kubernetes/Taskfile.yaml) for validation of cluster resources or continue to get familiar with the `kubectl` and `flux` CLI tools._
📍 _Run `task reconcile` to force Flux to sync your Git repository state_
```sh
task kubernetes:resources
flux check
flux get sources git flux-system
flux get ks -A
flux get hr -A
```
2. ⚠️ It might take `cert-manager` awhile to generate certificates, this is normal so be patient.
3. Check TCP connectivity to both the internal and external gateways:
3. 🏆 **Congratulations** if all goes smooth you will have a Kubernetes cluster managed by Flux and your Git repository is driving the state of your cluster.
📍 _The variables are only placeholders, replace them with your actual values_
4. 🧠 Now it's time to pause and go get some motel motor oil ☕ and admire you made it this far!
4. Check you can resolve DNS for `echo`, this should resolve to `${cloudflare_gateway_addr}`:
#### 🌐 Public DNS
📍 _The variables are only placeholders, replace them with your actual values_
The `external-dns` application created in the `networking` namespace will handle creating public DNS records. By default, `echo-server` and the `flux-webhook` are the only subdomains reachable from the public internet. In order to make additional applications public you must set set the correct ingress class name and ingress annotations like in the HelmRelease for `echo-server`.
5. Check the status of your wildcard `Certificate`:
`k8s_gateway` will provide DNS resolution to external Kubernetes resources (i.e. points of entry to the cluster) from any device that uses your home DNS server. For this to work, your home DNS server must be configured to forward DNS queries for `${bootstrap_cloudflare.domain}` to `${bootstrap_cloudflare.gateway_vip}` instead of the upstream DNS server(s) it normally uses. This is a form of **split DNS** (aka split-horizon DNS / conditional forwarding).
```sh
kubectl -n network describe certificates
```
### 🌐 Public DNS
> [!TIP]
> Below is how to configure a Pi-hole for split DNS. Other platforms should be similar.
> 1. Apply this file on the Pihole server while substituting the variables
> 3. Query an internal-only subdomain from your workstation (any `internal` class ingresses): `dig @${home-dns-server-ip} echo-server-internal.${bootstrap_cloudflare.domain}`. It should resolve to `${bootstrap_cloudflare.ingress_vip}`.
> Use the `envoy-external` gateway on `HTTPRoutes` to make applications public to the internet. These are also accessible on your private network once you set up split DNS.
If you're having trouble with DNS be sure to check out these two GitHub discussions: [Internal DNS](https://github.com/onedr0p/cluster-template/discussions/719) and [Pod DNS resolution broken](https://github.com/onedr0p/cluster-template/discussions/635).
The `external-dns` application created in the `network` namespace will handle creating public DNS records. By default, `echo` and the `flux-webhook` are the only subdomains reachable from the public internet. In order to make additional applications public you must **set the correct gateway** like in the HelmRelease for `echo`.
... Nothing working? That is expected, this is DNS after all!
### 🏠 Home DNS
#### 📜 Certificates
> [!TIP]
> Use the `envoy-internal` gateway on `HTTPRoutes` to make applications private to your network. If you're having trouble with internal DNS resolution check out [this](https://github.com/onedr0p/cluster-template/discussions/719) GitHub discussion.
By default this template will deploy a wildcard certificate using the Let's Encrypt **staging environment**, which prevents you from getting rate-limited by the Let's Encrypt production servers if your cluster doesn't deploy properly (for example due to a misconfiguration). Once you are sure you will keep the cluster up for more than a few hours be sure to switch to the production servers as outlined in `config.yaml`.
`k8s_gateway` will provide DNS resolution to external Kubernetes resources (i.e. points of entry to the cluster) from any device that uses your home DNS server. For this to work, your home DNS server must be configured to forward DNS queries for `${cloudflare_domain}` to `${cluster_dns_gateway_addr}` instead of the upstream DNS server(s) it normally uses. This is a form of **split DNS** (aka split-horizon DNS / conditional forwarding).
📍 _You will need a production certificate to reach internet-exposed applications through `cloudflared`._
_... Nothing working? That is expected, this is DNS after all!_
#### 🪝 Github Webhook
### 🪝 GitHub Webhook
By default Flux will periodically check your git repository for changes. Inorder to have Flux reconcile on `git push` you must configure Github to send `push` events to Flux.
By default Flux will periodically check your git repository for changes. In-order to have Flux reconcile on `git push` you must configure GitHub to send `push` events to Flux.
> [!NOTE]
> This will only work after you have switched over certificates to the Let's Encrypt Production servers.
1. Obtain the webhook path:
1. Obtain the webhook path
📍 _Hook id and path should look like `/hook/12ebd1e363c641dc3c2e430ecf3cee2b3c7a5ac9e1234506f6f5f3ce1230e123`_
📍 _Hook id and path should look like `/hook/12ebd1e363c641dc3c2e430ecf3cee2b3c7a5ac9e1234506f6f5f3ce1230e123`_
```sh
kubectl -n flux-system get receiver github-receiver -o jsonpath='{.status.webhookPath}'
kubectl -n flux-system get receiver github-webhook --output=jsonpath='{.status.webhookPath}'
```
2. Piece together the full URL with the webhook path appended
2. Piece together the full URL with the webhook path appended:
3. Navigate to the settings of your repository on Github, under "Settings/Webhooks" press the "Add webhook" button. Fill in the webhook URL and your `bootstrap_github_webhook_token` secret in `config.yaml`, Content type: `application/json`, Events: Choose Just the push event, and save.
3. Navigate to the settings of your repository on GitHub, under "Settings/Webhooks" press the "Add webhook" button. Fill in the webhook URL and your token from `github-push-token.txt`, Content type: `application/json`, Events: Choose Just the push event, and save.
## 💥 Reset
There might be a situation where you want to destroy your Kubernetes cluster. The following command will reset your nodes back to maintenance mode, append `--force` to completely format your the Talos installation. Either way the nodes should reboot after the command has run.
> [!CAUTION]
> **Resetting** the cluster **multiple times in a short period of time** could lead to being **rate limited by DockerHub or Let's Encrypt**.
There might be a situation where you want to destroy your Kubernetes cluster. The following command will reset your nodes back to maintenance mode.
```sh
task talos:reset # --force
task talos:reset
```
## 🛠️ Talos and Kubernetes Maintenance
#### ⚙️ Updating Talos node configuration
### ⚙️ Updating Talos node configuration
📍 _Ensure you have updated `talconfig.yaml` and any patches with your updated configuration._
> [!TIP]
> Ensure you have updated `talconfig.yaml` and any patches with your updated configuration. In some cases you **not only need to apply the configuration but also upgrade talos** to apply new configuration.
```sh
# (Re)generate the Talos config
task talos:generate-config
# Apply the config to the node
task talos:apply-config HOSTNAME=? MODE=?
# e.g. task talos:apply-config HOSTNAME=k8s-0 MODE=reboot
task talos:apply-node IP=? MODE=?
# e.g. task talos:apply-node IP=10.10.10.10 MODE=auto
```
#### ⬆️ Updating Talos and Kubernetes versions
### ⬆️ Updating Talos and Kubernetes versions
📍 _Ensure the `talosVersion` and `kubernetesVersion` in `talhelper.yaml` are up-to-date with the version you wish to upgrade to._
```sh
# Upgrade the whole cluster to a newer Talos version
task talos:upgrade-cluster
# e.g. task talos:upgrade-cluster
```
> [!TIP]
> Ensure the `talosVersion` and `kubernetesVersion` in `talenv.yaml` are up-to-date with the version you wish to upgrade to.
```sh
# Upgrade node to a newer Talos version
task talos:upgrade-node HOSTNAME=?
# e.g. task talos:upgrade HOSTNAME=k8s-0
task talos:upgrade-node IP=?
# e.g. task talos:upgrade-node IP=10.10.10.10
```
```sh
@@ -324,96 +300,183 @@ task talos:upgrade-k8s
# e.g. task talos:upgrade-k8s
```
### ➕ Adding a node to your cluster
At some point you might want to expand your cluster to run more workloads and/or improve the reliability of your cluster. Keep in mind it is recommended to have an **odd number** of control plane nodes for quorum reasons.
You don't need to re-bootstrap the cluster to add new nodes. Follow these steps:
1. **Prepare the new node**: Review the [Stage 2: Machine Preparation](#stage-2-machine-preparation) section and boot your new node into maintenance mode.
2. **Get the node information**: While the node is in maintenance mode, retrieve the disk and MAC address information needed for configuration:
```sh
talosctl get disks -n <ip> --insecure
talosctl get links -n <ip> --insecure
```
3. **Update the configuration**: Read the documentation for [talhelper](https://budimanjojo.github.io/talhelper/latest/) and extend the `talconfig.yaml` file manually with the new node information (including the disk and MAC address from step 2).
4. **Generate and apply the configuration**:
```sh
# Render your talosconfig based on the talconfig.yaml file
task talos:generate-config
# Apply the configuration to the node
task talos:apply-node IP=?
# e.g. task talos:apply-node IP=10.10.10.10
```
The node should join the cluster automatically and workloads will be scheduled once they report as ready.
## 🤖 Renovate
[Renovate](https://www.mend.io/renovate) is a tool that automates dependency management. It is designed to scan your repository around the clock and open PRs for out-of-date dependencies it finds. Common dependencies it can discover are Helm charts, container images, GitHub Actions, Ansible roles... even Flux itself! Merging a PR will cause Flux to apply the update to your cluster.
[Renovate](https://www.mend.io/renovate) is a tool that automates dependency management. It is designed to scan your repository around the clock and open PRs for out-of-date dependencies it finds. Common dependencies it can discover are Helm charts, container images, GitHub Actions and more! In most cases merging a PR will cause Flux to apply the update to your cluster.
To enable Renovate, click the 'Configure' button over at their [Github app page](https://github.com/apps/renovate) and select your repository. Renovate creates a "Dependency Dashboard" as an issue in your repository, giving an overview of the status of all updates. The dashboard has interactive checkboxes that let you do things like advance scheduling or reattempt update PRs you closed without merging.
The base Renovate configuration in your repository can be viewed at [.github/renovate.json5](./.github/renovate.json5). By default it is scheduled to be active with PRs every weekend, but you can [change the schedule to anything you want](https://docs.renovatebot.com/presets-schedule), or remove it if you want Renovate to open PRs right away.
The base Renovate configuration in your repository can be viewed at [.renovaterc.json5](.renovaterc.json5). By default it is scheduled to be active with PRs every weekend, but you can [change the schedule to anything you want](https://docs.renovatebot.com/presets-schedule), or remove it if you want Renovate to open PRs immediately.
## 🐛 Debugging
Below is a general guide on trying to debug an issue with an resource or application. For example, if a workload/resource is not showing up or a pod has started but in a `CrashLoopBackOff` or `Pending` state.
Below is a general guide on trying to debug an issue with an resource or application. For example, if a workload/resource is not showing up or a pod has started but in a `CrashLoopBackOff` or `Pending` state. These steps do not include a way to fix the problem as the problem could be one of many different things.
1. Start by checking all Flux Kustomizations & Git Repository & OCI Repository and verify they are healthy.
1. Check if the Flux resources are up-to-date and in a ready state:
📍 _Run `task reconcile` to force Flux to sync your Git repository state_
```sh
flux get sources oci -A
flux get sources git -A
flux get ks -A
```
2. Then check all the Flux Helm Releases and verify they are healthy.
```sh
flux get hr -A
```
3. Then check the if the pod is present.
2. Do you see the pod of the workload you are debugging:
```sh
kubectl -n <namespace> get pods -o wide
```
4. Then check the logs of the pod if its there.
3. Check the logs of the pod if it's there:
```sh
kubectl -n <namespace> logs <pod-name> -f
# or
stern -n <namespace> <fuzzy-name>
```
5. If a resource exists try to describe it to see what problems it might have.
4. If a resource exists, try to describe it to see what problems it might have:
```sh
kubectl -n <namespace> describe <resource> <name>
```
6. Check the namespace events
5. Check the namespace events:
```sh
kubectl -n <namespace> get events --sort-by='.metadata.creationTimestamp'
```
Resolving problems that you have could take some tweaking of your YAML manifests in order to get things working, other times it could be a external factor like permissions on NFS. If you are unable to figure out your problem see the help section below.
Resolving problems that you have could take some tweaking of your YAML manifests in order to get things working, other times it could be a external factor like permissions on a NFS server. If you are unable to figure out your problem see the support sections below.
## 👉 Help
## 🧹 Tidy up
- Make a post in this repository's Github [Discussions](https://github.com/onedr0p/cluster-template/discussions).
- Start a thread in the `#support` or `#cluster-template` channels in the [Home Operations](https://discord.gg/home-operations) Discord server.
Once your cluster is fully configured and you no longer need to run `task configure`, it's a good idea to clean up the repository by removing the [templates](./templates) directory and any files related to the templating process. This will help eliminate unnecessary clutter from the upstream template repository and resolve any "duplicate registry" warnings from Renovate.
1. Tidy up your repository:
```sh
task template:tidy
```
2. Push your changes to git:
```sh
git add -A
git commit -m "chore: tidy up :broom:"
git push
```
## ❔ What's next
The cluster is your oyster (or something like that). Below are some optional considerations you might want to review.
There's a lot to absorb here, especially if you're new to these tools. Take some time to familiarize yourself with the tooling and understand how all the components interconnect. Dive into the documentation of the various tools included — they are a valuable resource. This shouldn't be a production environment yet, so embrace the freedom to experiment. Move fast, break things intentionally, and challenge yourself to fix them.
### Ship it
To browse or get ideas on applications people are running, community member [@whazor](https://github.com/whazor) created [Kubesearch](https://kubesearch.dev) as a creative way to search Flux HelmReleases across Github and Gitlab.
Below are some optional considerations you may want to explore.
### DNS
Instead of using [k8s_gateway](https://github.com/ori-edge/k8s_gateway) to provide DNS for your applications you might want to check out [external-dns](https://github.com/kubernetes-sigs/external-dns), it has wide support for many different providers such as [Pi-hole](https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/pihole.md), [UniFi](https://github.com/kashalls/external-dns-unifi-webhook), [Adguard Home](https://github.com/muhlba91/external-dns-provider-adguard), [Bind](https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/rfc2136.md) and more.
The template uses [k8s_gateway](https://github.com/k8s-gateway/k8s_gateway) to provide DNS for your applications, consider exploring [external-dns](https://github.com/kubernetes-sigs/external-dns) as an alternative.
External-DNS offers broad support for various DNS providers, including but not limited to:
This flexibility allows you to integrate seamlessly with a range of DNS solutions to suit your environment and offload DNS from your cluster to your router, or external device.
### Secrets
SOPS is an excellent tool for managing secrets in a GitOps workflow. However, it can become cumbersome when rotating secrets or maintaining a single source of truth for secret items.
For a more streamlined approach to those issues, consider [External Secrets](https://external-secrets.io/latest/). This tool allows you to move away from SOPs and leverage an external provider for managing your secrets. External Secrets supports a wide range of providers, from cloud-based solutions to self-hosted options.
### Storage
The included CSI (openebs in local-hostpath mode) is a great start for storage but soon you might find you need more features like replicated block storage, or to connect to a NFS/SMB/iSCSI server. If you need any of those features be sure to check out the projects like [rook-ceph](https://github.com/rook/rook), [longhorn](https://github.com/longhorn/longhorn), [openebs](https://github.com/openebs/openebs), [democratic-csi](https://github.com/democratic-csi/democratic-csi), [csi-driver-nfs](https://github.com/kubernetes-csi/csi-driver-nfs),
and [synology-csi](https://github.com/SynologyOpenSource/synology-csi).
If your workloads require persistent storage with features like replication or connectivity to NFS, SMB, or iSCSI servers, there are several projects worth exploring:
These tools offer a variety of solutions to meet your persistent storage needs, whether you’re using cloud-native or self-hosted infrastructures.
### Community Repositories
Community member [@whazor](https://github.com/whazor) created [Kubesearch](https://kubesearch.dev) to allow searching Flux HelmReleases across Github and Gitlab repositories with the `kubesearch` topic.
## 🙋 Support
### Community
- Make a post in this repository's GitHub [Discussions](https://github.com/onedr0p/cluster-template/discussions).
- Start a thread in the `#support` or `#cluster-template` channels in the [Home Operations](https://discord.gg/home-operations) Discord server.
## 📺 Media
Check out these videos below. If you find them helpful, a like and subscribe goes a long way!
If this repo is too hot to handle or too cold to hold check out these following projects.
- [khuedoan/homelab](https://github.com/khuedoan/homelab) - _Modern self-hosting framework, fully automated from empty disk to operating services with a single command._
- [danmanners/aws-argo-cluster-template](https://github.com/danmanners/aws-argo-cluster-template) - _A community opinionated template for deploying Kubernetes clusters on-prem and in AWS using Pulumi, SOPS, Sealed Secrets, GitHub Actions, Renovate, Cilium and more!_
- [ricsanfre/pi-cluster](https://github.com/ricsanfre/pi-cluster) - _Pi Kubernetes Cluster. Homelab kubernetes cluster automated with Ansible and ArgoCD_
- [techno-tim/k3s-ansible](https://github.com/techno-tim/k3s-ansible) - _The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more_
- [ajaykumar4/cluster-template](https://github.com/ajaykumar4/cluster-template) - _A template for deploying a Talos Kubernetes cluster including Argo for GitOps_
- [khuedoan/homelab](https://github.com/khuedoan/homelab) - _Fully automated homelab from empty disk to running services with a single command._
- [mitchross/k3s-argocd-starter](https://github.com/mitchross/k3s-argocd-starter) - starter kit for k3s, argocd
- [ricsanfre/pi-cluster](https://github.com/ricsanfre/pi-cluster) - _Pi Kubernetes Cluster. Homelab kubernetes cluster automated with Ansible and FluxCD_
- [techno-tim/k3s-ansible](https://github.com/techno-tim/k3s-ansible) - _The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more. Build. Destroy. Repeat._
## ⭐ Stargazers
<div align="center">
[](https://star-history.com/#onedr0p/cluster-template&Date)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
