diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..504d836 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,15 @@ +# Contributing Guidelines + +## Reporting an issue + +* Create an issue describing the problem. + +## Submitting a Pull Request + +All commits require a [DCO](https://developercertificate.org/) sign-off. +This is done by committing with the `--signoff` flag. + +* Create an issue describing the problem and your proposed solution. +* Wait for feedback from the maintainers. +* Fork the repo, develop and test your code changes. +* Submit a pull request. diff --git a/LICENSE b/LICENSE index db88f25..18759f7 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2021 Serge +Copyright (c) 2021 Serge Logvinov Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index 64a7f45..9c7fb64 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,36 @@ # Terraform examples to launch Talos. -I store here the terraform code to launch Talos in the clouds. -I wouldn't use the terrafrom modules from internet. -The goal is to create all cloud services from scratch. +This repository was created to store Terraform code for launching Talos in the clouds/bare-metal. +When I added features/cloud platform integrations to Talos, I needed to run tests manually. +To make things easier, I created this repository. + +There are no GitOps best practices here - no FluxCD, ArgoCD, or other GitOps tools. +Each step is applied manually because I need to test everything to ensure it works as expected. + +* I chose not to use Terraform modules from the internet; the goal here is to build all cloud services from scratch. +* I `don’t maintain backward compatibility` and always use the latest versions of Terraform and cloud provider tools. +* Kubernetes isn’t fully ready for multi-cloud environments, as many components were designed for single-environment setups. So did some changes to each cloud provider controllers to improve compatibility. (like CCM, CSI, etc.) +* The [Talos CCM](https://github.com/siderolabs/talos-cloud-controller-manager) project was created to make multi-cloud setups more cloud-native, addressing some common issues in multi-cloud environments. + +Some examples are production ready, and I’ve been using them with minor adjustments to fit company’s needs. +In most cases in my production setup, I use two or more cloud providers within a single Kubernetes cluster. + +Everything here is under the `MIT license`. +Feel free to clone, copy the code. +If this project helps you, please give it a `star`. +It helps me to understand how many people are interested in this project/ideas. +And it motivates me to keep working on it. Your support encourages me to add/sync new features. ## Ideas -First, I will create separate clusters on each cloud provider, test them thoroughly, and bring them close to production readiness. When I merge these separate Kubernetes clusters into one, they will have a single control plane. +First, I will create separate clusters on each cloud provider, test them thoroughly, and bring them close to production readiness. +When I merge these separate Kubernetes clusters into one, they will have a single control plane. Why is it so important? Having a single Kubernetes control plane that spans multiple cloud providers can offer several benefits: -* Improved resilience and availability: By using multiple cloud providers, you can reduce the risk of downtime due to a single point of failure. +* Improved resilience and availability: By using multiple cloud providers, you can reduce the risk of downtime due to cloud provider outages or other issues. * Flexibility: A single control plane allows you to easily move workloads between different cloud providers, depending on your needs. * Cost savings: You can take advantage of the different pricing models and discounts offered by different cloud providers to save on costs. * Improved security: By using multiple cloud providers, you can implement a defense-in-depth strategy to protect your data and reduce the risk of a security breach. @@ -20,16 +38,16 @@ Having a single Kubernetes control plane that spans multiple cloud providers can ## Clouds -| Platform | Checked Talos version | Addons | Setup type | Nat-IPv4 | IPv6 | Pod with IPv6 | +| Platform | Checked Talos version | Addons | Setup type | Nat-IPv4 | IPv6 | Pod with global IPv6 | |---|---|---|---|---|---|---| -| [Azure](azure) | 1.3.4 | CCM,CSI,Autoscaler | many regions, many zones | ✓ | ✓ | | +| [Azure](azure) | 1.3.4 | CCM,CSI,Autoscaler | many regions, many zones | ✓ | ✓ | ✗ | | [Exoscale](exoscale) | 1.3.0 | CCM,Autoscaler | many regions | ✗ | | | | [GCP](gcp-zonal) | 1.3.4 | CCM,CSI,Autoscaler | one region, many zones | ✓ | ✓ | ✓ | | [Hetzner](hetzner) | 1.7.6 | CCM,CSI,Autoscaler | many regions, one network zone | ✗ | ✓ | ✓ | | [Openstack](openstack) | 1.3.4 | CCM,CSI | many regions, many zones | ✓ | ✓ | ✓ | -| [Oracle](oracle) | 1.3.4 | CCM,CSI,Autoscaler | one region, many zones | ✓ | ✓ | +| [Oracle](oracle) | 1.3.4 | CCM,CSI,Autoscaler | one region, many zones | ✓ | ✓ | | | [Proxmox](proxmox) | 1.7.6 | CCM,CSI | one region, mny zones | ✓ | ✓ | ✓ | -| [Scaleway](scaleway) | 1.7.6 | CCM,CSI | one region | ✓ | ✓ | | +| [Scaleway](scaleway) | 1.7.6 | CCM,CSI | one region | ✓ | ✓ | ✓ | ## Known issues @@ -37,10 +55,12 @@ Having a single Kubernetes control plane that spans multiple cloud providers can ## Multi cloud compatibility -CCM compatibility, which can work together: +CCM controllers have different modes: * Talos CCM in mode: `cloud-node` * Other CCMs in mode: `cloud-node-lifecycle` +CCM compatibility has been tested in multi-cloud setups, and in most cases, they work well together. + | | Azure | GCP | Hetzner | Openstack | Proxmox | |---|---|---|---|---|---| | Azure | | ✓ | ✓ | ✓ | ✓ | @@ -60,3 +80,10 @@ And I can disable conntrack too. * **coredns-local** (daemonsets) uses dummy interface on al nodes and has ip ```169.254.2.53``` It decrease the dns response (all traffic does not leave the node). * **rancher.io/local-path** as default storage class. + +The common deployoment you can find in [_deployments](/_deployments/) folder. + +## References + +* [Talos](https://www.talos.dev/) +* [Talos CCM](https://github.com/siderolabs/talos-cloud-controller-manager) diff --git a/_old/prepare/auth.tf b/_old/prepare/auth.tf deleted file mode 100644 index 0615495..0000000 --- a/_old/prepare/auth.tf +++ /dev/null @@ -1,4 +0,0 @@ - -provider "hcloud" { - token = var.hcloud_token -} diff --git a/_old/prepare/main.tf b/_old/prepare/main.tf deleted file mode 100644 index 11d2c60..0000000 --- a/_old/prepare/main.tf +++ /dev/null @@ -1,50 +0,0 @@ - -resource "hcloud_ssh_key" "snapshot" { - name = "Snapshoter" - public_key = file("~/.ssh/terraform.pub") - labels = merge(var.tags, { type = "infra" }) -} - -resource "hcloud_server" "talos" { - location = element(var.regions, 1) - name = "talos-os" - image = "debian-10" - rescue = "linux64" - server_type = "cx11" - keep_disk = true - backups = false - ssh_keys = [hcloud_ssh_key.snapshot.id] - firewall_ids = [] - labels = merge(var.tags, { type = "infra", label = "template" }) - - lifecycle { - ignore_changes = [ - firewall_ids, - status, - ssh_keys, - ] - } - - connection { - user = "root" - private_key = file("~/.ssh/terraform") - host = self.ipv4_address - timeout = "10m" - } - provisioner "remote-exec" { - inline = [ - "apt-get install -y wget", - "wget -O /tmp/openstack.tar.gz https://github.com/talos-systems/talos/releases/download/${var.talos_version}/openstack-amd64.tar.gz", - "cd /tmp && tar xzf /tmp/openstack.tar.gz && dd if=/tmp/disk.raw of=/dev/sda && sync", - "mount /dev/sda3 /mnt && sed -i 's/set timeout=3/set timeout=10/g' /mnt/grub/grub.cfg && umount /mnt", - "shutdown -h now" - ] - } -} - -resource "hcloud_snapshot" "talos" { - server_id = hcloud_server.talos.id - description = "talos system disk" - labels = merge(var.tags, { type = "infra" }) - depends_on = [hcloud_server.talos] -} diff --git a/_old/prepare/variables.tf b/_old/prepare/variables.tf deleted file mode 100644 index d2193ee..0000000 --- a/_old/prepare/variables.tf +++ /dev/null @@ -1,26 +0,0 @@ - -variable "hcloud_token" { - description = "The hezner cloud token (export TF_VAR_hcloud_token=$TOKEN)" - type = string - sensitive = true -} - -variable "regions" { - description = "The id of the hezner region (oreder is important)" - type = list(string) - default = ["nbg1"] -} - -variable "tags" { - description = "Tags of resources" - type = map(string) - default = { - environment = "Develop" - } -} - -variable "talos_version" { - description = "Talos image version" - type = string - default = "v0.10.0" -} diff --git a/_old/prepare/versions.tf b/_old/prepare/versions.tf deleted file mode 100644 index 1862469..0000000 --- a/_old/prepare/versions.tf +++ /dev/null @@ -1,9 +0,0 @@ -terraform { - required_providers { - hcloud = { - source = "hetznercloud/hcloud" - version = "~> 1.26.2" - } - } - required_version = ">= 0.15" -} diff --git a/aws/README.md b/aws/README.md new file mode 100644 index 0000000..06f30c9 --- /dev/null +++ b/aws/README.md @@ -0,0 +1,3 @@ +# AWS + +Status: **abandoned** diff --git a/digitalocean/README.md b/digitalocean/README.md new file mode 100644 index 0000000..dad63fd --- /dev/null +++ b/digitalocean/README.md @@ -0,0 +1,3 @@ +# Digitalocean + +Status: **abandoned** diff --git a/system_os/digitalocean/Makefile b/digitalocean/images/Makefile similarity index 100% rename from system_os/digitalocean/Makefile rename to digitalocean/images/Makefile diff --git a/system_os/digitalocean/do.pkr.hcl b/digitalocean/images/do.pkr.hcl similarity index 100% rename from system_os/digitalocean/do.pkr.hcl rename to digitalocean/images/do.pkr.hcl diff --git a/system_os/digitalocean/variables.pkr.hcl b/digitalocean/images/variables.pkr.hcl similarity index 100% rename from system_os/digitalocean/variables.pkr.hcl rename to digitalocean/images/variables.pkr.hcl diff --git a/proxmox/deployments/talos-ccm.yaml b/proxmox/deployments/talos-ccm.yaml index 7ea1392..836ca33 100644 --- a/proxmox/deployments/talos-ccm.yaml +++ b/proxmox/deployments/talos-ccm.yaml @@ -1,7 +1,8 @@ image: - repository: ghcr.io/sergelogvinov/talos-cloud-controller-manager - tag: nodeipam + # repository: ghcr.io/sergelogvinov/talos-cloud-controller-manager + pullPolicy: Always + tag: edge service: containerPort: 50258 @@ -14,6 +15,7 @@ logVerbosityLevel: 4 enabledControllers: - cloud-node + - node-csr-approval - node-ipam-controller extraArgs: diff --git a/proxmox/variables.tf b/proxmox/variables.tf index e496432..b2291f9 100644 --- a/proxmox/variables.tf +++ b/proxmox/variables.tf @@ -38,7 +38,7 @@ variable "vpc_main_cidr" { variable "release" { type = string description = "The version of the Talos image" - default = "1.8.0" + default = "1.7.6" } data "sops_file" "tfvars" { diff --git a/system_os/gcp/gcp.pkr.hcl b/system_os/gcp/gcp.pkr.hcl deleted file mode 100644 index 8f5d1e0..0000000 --- a/system_os/gcp/gcp.pkr.hcl +++ /dev/null @@ -1,62 +0,0 @@ - -packer { - required_plugins { - googlecompute = { - version = ">= 1.0.0" - source = "github.com/hashicorp/googlecompute" - } - } -} - -variable "google_account" { - type = string - default = "" - sensitive = true -} - -variable "google_project" { - type = string - default = "" - sensitive = true -} - -variable "google_locations" { - type = list(string) - default = ["europe-west4"] - sensitive = true -} - -variable "talos_version" { - type = string - default = "v1.3.3" -} - -source "googlecompute" "talos" { - account_file = var.google_account - project_id = var.google_project - zone = "europe-west4-a" - subnetwork = "default" - source_image_family = "debian-10" - ssh_username = "debian" - - machine_type = "e2-small" - disk_size = 10 - disk_type = "pd-standard" - - image_name = "talos" - image_description = "talos system disk" - image_family = "talos" - image_licenses = ["projects/vm-options/global/licenses/enable-vmx"] - image_storage_locations = var.google_locations -} - -build { - sources = ["source.googlecompute.talos"] - provisioner "shell" { - inline = [ - "sudo apt-get install -y wget", - "wget -O /tmp/talos.tar.gz https://github.com/talos-systems/talos/releases/download/${var.talos_version}/gcp-amd64.tar.gz", - "tar xOzf /tmp/talos.tar.gz | sudo dd of=/dev/sda", - ] - } -}