update readme

CONTRIBUTING.md

@@ -0,0 +1,15 @@
+# Contributing Guidelines
+
+## Reporting an issue
+
+* Create an issue describing the problem.
+
+## Submitting a Pull Request
+
+All commits require a [DCO](https://developercertificate.org/) sign-off.
+This is done by committing with the `--signoff` flag.
+
+* Create an issue describing the problem and your proposed solution.
+* Wait for feedback from the maintainers.
+* Fork the repo, develop and test your code changes.
+* Submit a pull request.

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2021 Serge
+Copyright (c) 2021 Serge Logvinov
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,18 +1,36 @@
 # Terraform examples to launch Talos.
 
-I store here the terraform code to launch Talos in the clouds.
-I wouldn't use the terrafrom modules from internet.
-The goal is to create all cloud services from scratch.
+This repository was created to store Terraform code for launching Talos in the clouds/bare-metal.
+When I added features/cloud platform integrations to Talos, I needed to run tests manually.
+To make things easier, I created this repository.
+
+There are no GitOps best practices here - no FluxCD, ArgoCD, or other GitOps tools.
+Each step is applied manually because I need to test everything to ensure it works as expected.
+
+* I chose not to use Terraform modules from the internet; the goal here is to build all cloud services from scratch.
+* I `don’t maintain backward compatibility` and always use the latest versions of Terraform and cloud provider tools.
+* Kubernetes isn’t fully ready for multi-cloud environments, as many components were designed for single-environment setups. So did some changes to each cloud provider controllers to improve compatibility. (like CCM, CSI, etc.)
+* The [Talos CCM](https://github.com/siderolabs/talos-cloud-controller-manager) project was created to make multi-cloud setups more cloud-native, addressing some common issues in multi-cloud environments.
+
+Some examples are production ready, and I’ve been using them with minor adjustments to fit company’s needs.
+In most cases in my production setup, I use two or more cloud providers within a single Kubernetes cluster.
+
+Everything here is under the `MIT license`.
+Feel free to clone, copy the code.
+If this project helps you, please give it a `star`.
+It helps me to understand how many people are interested in this project/ideas.
+And it motivates me to keep working on it. Your support encourages me to add/sync new features.
 
 ## Ideas
 
-First, I will create separate clusters on each cloud provider, test them thoroughly, and bring them close to production readiness. When I merge these separate Kubernetes clusters into one, they will have a single control plane.
+First, I will create separate clusters on each cloud provider, test them thoroughly, and bring them close to production readiness.
+When I merge these separate Kubernetes clusters into one, they will have a single control plane.
 
 Why is it so important?
 
 Having a single Kubernetes control plane that spans multiple cloud providers can offer several benefits:
 
-* Improved resilience and availability: By using multiple cloud providers, you can reduce the risk of downtime due to a single point of failure.
+* Improved resilience and availability: By using multiple cloud providers, you can reduce the risk of downtime due to cloud provider outages or other issues.
 * Flexibility: A single control plane allows you to easily move workloads between different cloud providers, depending on your needs.
 * Cost savings: You can take advantage of the different pricing models and discounts offered by different cloud providers to save on costs.
 * Improved security: By using multiple cloud providers, you can implement a defense-in-depth strategy to protect your data and reduce the risk of a security breach.
@@ -20,16 +38,16 @@ Having a single Kubernetes control plane that spans multiple cloud providers can
 
 ## Clouds
 
-| Platform | Checked Talos version | Addons | Setup type | Nat-IPv4 | IPv6 | Pod with IPv6 |
+| Platform | Checked Talos version | Addons | Setup type | Nat-IPv4 | IPv6 | Pod with global IPv6 |
 |---|---|---|---|---|---|---|
-| [Azure](azure)         | 1.3.4  | CCM,CSI,Autoscaler | many regions, many zones | ✓ | ✓ | |
+| [Azure](azure)         | 1.3.4  | CCM,CSI,Autoscaler | many regions, many zones | ✓ | ✓ | ✗ |
 | [Exoscale](exoscale)   | 1.3.0  | CCM,Autoscaler     | many regions | ✗ | | |
 | [GCP](gcp-zonal)       | 1.3.4  | CCM,CSI,Autoscaler | one region, many zones | ✓ | ✓ | ✓ |
 | [Hetzner](hetzner)     | 1.7.6  | CCM,CSI,Autoscaler | many regions, one network zone | ✗ | ✓ | ✓ |
 | [Openstack](openstack) | 1.3.4  | CCM,CSI            | many regions, many zones | ✓ | ✓ | ✓ |
-| [Oracle](oracle)       | 1.3.4  | CCM,CSI,Autoscaler | one region, many zones | ✓ | ✓ |
+| [Oracle](oracle)       | 1.3.4  | CCM,CSI,Autoscaler | one region, many zones | ✓ | ✓ | |
 | [Proxmox](proxmox)     | 1.7.6  | CCM,CSI            | one region, mny zones | ✓ | ✓ | ✓ |
-| [Scaleway](scaleway)   | 1.7.6  | CCM,CSI            | one region | ✓ | ✓ | |
+| [Scaleway](scaleway)   | 1.7.6  | CCM,CSI            | one region | ✓ | ✓ | ✓ |
 
 ## Known issues
 
@@ -37,10 +55,12 @@ Having a single Kubernetes control plane that spans multiple cloud providers can
 
 ## Multi cloud compatibility
 
-CCM compatibility, which can work together:
+CCM controllers have different modes:
 * Talos CCM in mode: `cloud-node`
 * Other CCMs in mode: `cloud-node-lifecycle`
 
+CCM compatibility has been tested in multi-cloud setups, and in most cases, they work well together.
+
 |   | Azure | GCP | Hetzner | Openstack | Proxmox |
 |---|---|---|---|---|---|
 | Azure     | | ✓ | ✓ | ✓ | ✓ |
@@ -60,3 +80,10 @@ And I can disable conntrack too.
 * **coredns-local** (daemonsets) uses dummy interface on al nodes and has ip ```169.254.2.53```
 It decrease the dns response (all traffic does not leave the node).
 * **rancher.io/local-path** as default storage class.
+
+The common deployoment you can find in [_deployments](/_deployments/) folder.
+
+## References
+
+* [Talos](https://www.talos.dev/)
+* [Talos CCM](https://github.com/siderolabs/talos-cloud-controller-manager)

_old/prepare/auth.tf

@@ -1,4 +0,0 @@
-
-provider "hcloud" {
-  token = var.hcloud_token
-}

_old/prepare/main.tf

@@ -1,50 +0,0 @@
-
-resource "hcloud_ssh_key" "snapshot" {
-  name       = "Snapshoter"
-  public_key = file("~/.ssh/terraform.pub")
-  labels     = merge(var.tags, { type = "infra" })
-}
-
-resource "hcloud_server" "talos" {
-  location     = element(var.regions, 1)
-  name         = "talos-os"
-  image        = "debian-10"
-  rescue       = "linux64"
-  server_type  = "cx11"
-  keep_disk    = true
-  backups      = false
-  ssh_keys     = [hcloud_ssh_key.snapshot.id]
-  firewall_ids = []
-  labels       = merge(var.tags, { type = "infra", label = "template" })
-
-  lifecycle {
-    ignore_changes = [
-      firewall_ids,
-      status,
-      ssh_keys,
-    ]
-  }
-
-  connection {
-    user        = "root"
-    private_key = file("~/.ssh/terraform")
-    host        = self.ipv4_address
-    timeout     = "10m"
-  }
-  provisioner "remote-exec" {
-    inline = [
-      "apt-get install -y wget",
-      "wget -O /tmp/openstack.tar.gz https://github.com/talos-systems/talos/releases/download/${var.talos_version}/openstack-amd64.tar.gz",
-      "cd /tmp && tar xzf /tmp/openstack.tar.gz && dd if=/tmp/disk.raw of=/dev/sda && sync",
-      "mount /dev/sda3 /mnt && sed -i 's/set timeout=3/set timeout=10/g' /mnt/grub/grub.cfg && umount /mnt",
-      "shutdown -h now"
-    ]
-  }
-}
-
-resource "hcloud_snapshot" "talos" {
-  server_id   = hcloud_server.talos.id
-  description = "talos system disk"
-  labels      = merge(var.tags, { type = "infra" })
-  depends_on  = [hcloud_server.talos]
-}

_old/prepare/variables.tf

@@ -1,26 +0,0 @@
-
-variable "hcloud_token" {
-  description = "The hezner cloud token (export TF_VAR_hcloud_token=$TOKEN)"
-  type        = string
-  sensitive   = true
-}
-
-variable "regions" {
-  description = "The id of the hezner region (oreder is important)"
-  type        = list(string)
-  default     = ["nbg1"]
-}
-
-variable "tags" {
-  description = "Tags of resources"
-  type        = map(string)
-  default = {
-    environment = "Develop"
-  }
-}
-
-variable "talos_version" {
-  description = "Talos image version"
-  type        = string
-  default     = "v0.10.0"
-}

_old/prepare/versions.tf

@@ -1,9 +0,0 @@
-terraform {
-  required_providers {
-    hcloud = {
-      source  = "hetznercloud/hcloud"
-      version = "~> 1.26.2"
-    }
-  }
-  required_version = ">= 0.15"
-}

aws/README.md

@@ -0,0 +1,3 @@
+# AWS
+
+Status: **abandoned**

digitalocean/README.md

@@ -0,0 +1,3 @@
+# Digitalocean
+
+Status: **abandoned**

proxmox/deployments/talos-ccm.yaml

@@ -1,7 +1,8 @@
 
 image:
-  repository: ghcr.io/sergelogvinov/talos-cloud-controller-manager
-  tag: nodeipam
+  # repository: ghcr.io/sergelogvinov/talos-cloud-controller-manager
+  pullPolicy: Always
+  tag: edge
 
 service:
   containerPort: 50258
@@ -14,6 +15,7 @@ logVerbosityLevel: 4
 
 enabledControllers:
   - cloud-node
+  - node-csr-approval
   - node-ipam-controller
 
 extraArgs:

proxmox/variables.tf

@@ -38,7 +38,7 @@ variable "vpc_main_cidr" {
 variable "release" {
   type        = string
   description = "The version of the Talos image"
-  default     = "1.8.0"
+  default     = "1.7.6"
 }
 
 data "sops_file" "tfvars" {

system_os/gcp/gcp.pkr.hcl

@@ -1,62 +0,0 @@
-
-packer {
-  required_plugins {
-    googlecompute = {
-      version = ">= 1.0.0"
-      source  = "github.com/hashicorp/googlecompute"
-    }
-  }
-}
-
-variable "google_account" {
-  type      = string
-  default   = ""
-  sensitive = true
-}
-
-variable "google_project" {
-  type      = string
-  default   = ""
-  sensitive = true
-}
-
-variable "google_locations" {
-  type      = list(string)
-  default   = ["europe-west4"]
-  sensitive = true
-}
-
-variable "talos_version" {
-  type    = string
-  default = "v1.3.3"
-}
-
-source "googlecompute" "talos" {
-  account_file        = var.google_account
-  project_id          = var.google_project
-  zone                = "europe-west4-a"
-  subnetwork          = "default"
-  source_image_family = "debian-10"
-  ssh_username        = "debian"
-
-  machine_type = "e2-small"
-  disk_size    = 10
-  disk_type    = "pd-standard"
-
-  image_name              = "talos"
-  image_description       = "talos system disk"
-  image_family            = "talos"
-  image_licenses          = ["projects/vm-options/global/licenses/enable-vmx"]
-  image_storage_locations = var.google_locations
-}
-
-build {
-  sources = ["source.googlecompute.talos"]
-  provisioner "shell" {
-    inline = [
-      "sudo apt-get install -y wget",
-      "wget -O /tmp/talos.tar.gz https://github.com/talos-systems/talos/releases/download/${var.talos_version}/gcp-amd64.tar.gz",
-      "tar xOzf /tmp/talos.tar.gz | sudo dd of=/dev/sda",
-    ]
-  }
-}