From febe56224cab75ada71bb12fe7994c46f0a29aeb Mon Sep 17 00:00:00 2001 From: Andrei Kvapil Date: Thu, 8 Feb 2024 06:50:50 +0100 Subject: [PATCH] upd readmr Signed-off-by: Andrei Kvapil --- README.md | 156 ++++++++++++------ TODO | 30 ---- img/cozystack-logo.svg | 51 ++++++ manifests/cozystack-installer.yaml | 4 +- packages/apps/tenant/README.md | 47 ++++++ packages/core/installer/images/installer.json | 6 +- teststand/proxmox/haproxy.cfg | 37 ----- teststand/proxmox/nftables.conf | 46 ------ 8 files changed, 210 insertions(+), 167 deletions(-) delete mode 100644 TODO create mode 100644 img/cozystack-logo.svg create mode 100644 packages/apps/tenant/README.md delete mode 100644 teststand/proxmox/haproxy.cfg delete mode 100644 teststand/proxmox/nftables.conf diff --git a/README.md b/README.md index fac84d9f..0f4a70b4 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,72 @@ +![Cozystack](img/cozystack-logo.svg) + +[![Open Source](https://img.shields.io/badge/Open-Source-brightgreen)](https://opensource.org/) +[![GPLv3 License](https://img.shields.io/github/license/aenix.io/cozystack)](https://opensource.org/licenses/) +[![Support](https://img.shields.io/badge/$-support-12a0df.svg?style=flat)](https://aenix.io/contact-us/#meet) +[![Active](http://img.shields.io/badge/Status-Active-green.svg)](https://aenix.io/cozystack/) +[![GitHub Release](https://img.shields.io/github/release/aenix.io/cozystack.svg?style=flat)](https://github.com/aenix.io/cozystack) +[![GitHub Commit](https://img.shields.io/github/commit-activity/y/linbit/aenix.io/cozystack)](https://github.com/aenix.io/cozystack) + # Cozystack +**Cozystack** is an open-source **PaaS platform** for cloud providers. + +With Cozystack, you can transform your bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease. + +You can use Cozystack to build your own cloud or to provide a cost-effective development environments. + +## Use-Cases + +### As a backend for a public cloud + +Cozystack positions itself as a kind of framework for building public clouds. The key word here is framework. In this case, it's important to understand that Cozystack is made for cloud providers, not for end users. + +Despite having a graphical interface, the current security model does not imply public user access to your management cluster. + +Instead, end users get access to their own Kubernetes clusters, can order LoadBalancers and additional services from it, but they have no access and know nothing about your management cluster powered by Cozystack. + +Thus, to integrate with your billing system, it's enough to teach your system to go to the management Kubernetes and place a YAML file signifying the service you're interested in. Cozystack will do the rest of the work for you. + +![](https://aenix.io/wp-content/uploads/2024/02/Wireframe-1.png) + +### As a private cloud for Infrastructure-as-Code + +One of the use cases is a self-portal for users within your company, where they can order the service they're interested in or a managed database. + +You can implement best GitOps practices, where users will launch their own Kubernetes clusters and databases for their needs with a simple commit of configuration into your infrastructure Git repository. + +Thanks to the standardization of the approach to deploying applications, you can expand the platform's capabilities using the functionality of standard Helm charts. + +### As a Kubernetes distribution for Bare Metal + +We created Cozystack primarily for our own needs, having vast experience in building reliable systems on bare metal infrastructure. This experience led to the formation of a separate boxed product, which is aimed at standardizing and providing a ready-to-use tool for managing your infrastructure. + +Currently, Cozystack already solves a huge scope of infrastructure tasks: starting from provisioning bare metal servers, having a ready monitoring system, fast and reliable storage, a network fabric with the possibility of interconnect with your infrastructure, the ability to run virtual machines, databases, and much more right out of the box. + +All this makes Cozystack a convenient platform for delivering and launching your application on Bare Metal. + +## Screenshot + +![](https://aenix.io/wp-content/uploads/2023/12/cozystack1-1.png) + +## Core values + +### Standardization and unification +All components of the platform are based on open source tools and technologies which are widely known in the industry. + +### Collaborate, not compete +If a feature being developed for the platform could be useful to a upstream project, it should be contributed to upstream project, rather than being implemented within the platform. + +### API-first +Cozystack is based on Kubernetes and involves close interaction with its API. We don't aim to completely hide the all elements behind a pretty UI or any sort of customizations; instead, we provide a standard interface and teach users how to work with basic primitives. The web interface is used solely for deploying applications and quickly diving into basic concepts of platform. + ## Quick Start -Install dependicies: - -- `docker` -- `talosctl` -- `dialog` -- `nmap` -- `make` -- `yq` -- `kubectl` -- `helm` - ### Preapre infrastructure + +![](https://aenix.io/wp-content/uploads/2024/02/Wireframe-2.png) + You need 3 physical servers or VMs with nested virtualisation: ``` @@ -30,18 +82,29 @@ Any Linux system installed on it (eg. Ubuntu should be enough) **Note:** The VM should support `x86-64-v2` architecture, the most probably you can achieve this by setting cpu model to `host` +#### Install dependicies: + +- `docker` +- `talosctl` +- `dialog` +- `nmap` +- `make` +- `yq` +- `kubectl` +- `helm` + ### Netboot server Start matchbox with prebuilt Talos image for Cozystack: -``` +```bash sudo docker run --name=matchbox -d --net=host ghcr.io/aenix-io/cozystack/matchbox:v0.0.1 \ -address=:8080 \ -log-level=debug ``` Start DHCP-Server: -``` +```bash sudo docker run --name=dnsmasq -d --cap-add=NET_ADMIN --net=host quay.io/poseidon/dnsmasq \ -d -q -p0 \ --dhcp-range=192.168.100.3,192.168.100.254 \ @@ -57,7 +120,7 @@ sudo docker run --name=dnsmasq -d --cap-add=NET_ADMIN --net=host quay.io/poseido --dhcp-match=set:efi64,option:client-arch,9 \ --dhcp-boot=tag:efi64,ipxe.efi \ --dhcp-userclass=set:ipxe,iPXE \ - --dhcp-boot=tag:ipxe,http://192.168.100.250:8080/boot.ipxe \ + --dhcp-boot=tag:ipxe,http://192.168.100.254:8080/boot.ipxe \ --log-queries \ --log-dhcp ``` @@ -65,7 +128,7 @@ sudo docker run --name=dnsmasq -d --cap-add=NET_ADMIN --net=host quay.io/poseido Where: - `192.168.100.3,192.168.100.254` range to allocate IPs from - `192.168.100.1` your gateway -- `192.168.100.250` is address of your management server +- `192.168.100.254` is address of your management server Check status of containers: @@ -73,9 +136,9 @@ Check status of containers: docker ps ``` -Example output: +example output: -``` +```console CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 22044f26f74d quay.io/poseidon/dnsmasq "/usr/sbin/dnsmasq -…" 6 seconds ago Up 5 seconds dnsmasq 231ad81ff9e0 ghcr.io/aenix-io/cozystack/matchbox:v0.0.1 "/matchbox -address=…" 58 seconds ago Up 57 seconds matchbox @@ -143,22 +206,22 @@ EOT Run [talos-bootstrap](https://github.com/aenix-io/talos-bootstrap/) to deploy cluster: -``` +```bash talos-bootstrap install ``` Save admin kubeconfig to access your Kubernetes cluster: -``` +```bash cp -i kubeconfig ~/.kube/config ``` Check connection: -``` +```bash kubectl get ns ``` example output: -``` +```console NAME STATUS AGE default Active 7m56s kube-node-lease Active 7m56s @@ -191,25 +254,24 @@ EOT Create namesapce and install Cozystack system components: -``` +```bash kubectl create ns cozy-system kubectl apply -f cozystack-config.yaml kubectl apply -f manifests/cozystack-installer.yaml ``` (optional) You can check logs of installer: -``` +```bash kubectl logs -n cozy-system deploy/cozystack ``` Wait for a while, then check the status of installation: -``` +```bash kubectl get hr -A ``` Wait until all releases become to `Ready` state: - -``` +```console NAMESPACE NAME AGE READY STATUS cozy-cert-manager cert-manager 2m54s True Release reconciliation succeeded cozy-cert-manager cert-manager-issuers 2m54s True Release reconciliation succeeded @@ -241,18 +303,18 @@ tenant-root tenant-root 2m54s True Rel #### Configure Storage Setup alias to access LINSTOR: -``` +```bash alias linstor='kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor' ``` list your nodes -``` +```bash linstor node list ``` example output: -``` +```console +-------------------------------------------------------+ | Node | NodeType | Addresses | State | |=======================================================| @@ -264,13 +326,12 @@ example output: list empty devices: -``` +```bash linstor physical-storage list ``` example output: - -``` +```console +-------------------------------------------+ | Size | Rotational | Nodes | |===========================================| @@ -283,7 +344,7 @@ example output: create storage pools: -``` +```bash linstor ps cdp lvm srv1 /dev/sdb --pool-name data --storage-pool data linstor ps cdp lvm srv2 /dev/sdb --pool-name data --storage-pool data linstor ps cdp lvm srv3 /dev/sdb --pool-name data --storage-pool data @@ -291,13 +352,13 @@ linstor ps cdp lvm srv3 /dev/sdb --pool-name data --storage-pool data list storage pools: -``` +```bash linstor sp l ``` example output: -``` +```console +-------------------------------------------------------------------------------------------------------------------------------------+ | StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName | |=====================================================================================================================================| @@ -350,13 +411,12 @@ EOT list storageclasses: -``` +```bash kubectl get storageclasses ``` example output: - -``` +```console NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local (default) linstor.csi.linbit.com Delete WaitForFirstConsumer true 11m replicated linstor.csi.linbit.com Delete WaitForFirstConsumer true 11m @@ -369,7 +429,7 @@ To access your services select the range of unused IPs, eg. `192.168.100.200-192 **Note:** These IPs should be from the same network as nodes or they should have all necessary routes for them. Configure MetalLB to use and announce this range: -``` +```yaml kubectl create -f- < 28m data-etcd-1 Bound pvc-b198f493-fb47-431c-a7aa-3befcf38a7d2 10Gi RWO local 28m @@ -445,13 +505,12 @@ vmstorage-db-vmstorage-shortterm-1 Bound pvc-d8d9da02-523e-4ec7-809a-bf Check all pods are running: -``` +```bash kubectl get pod -n tenant-root ``` example output: - -``` +```console NAME READY STATUS RESTARTS AGE etcd-0 1/1 Running 0 90s etcd-1 1/1 Running 0 90s @@ -484,8 +543,7 @@ kubectl get svc -n tenant-root root-ingress-controller ``` example output: - -``` +```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE root-ingress-controller LoadBalancer 10.96.101.234 192.168.100.200 80:31879/TCP,443:31262/TCP 49s ``` @@ -495,6 +553,6 @@ Use `grafana.example.org` (under 192.168.100.200) to access system monitoring, w - login: `admin` - password: -``` +```bash kubectl get secret -n tenant-root grafana-admin-password -o go-template='{{ printf "%s\n" (index .data "password" | base64decode) }}' ``` diff --git a/TODO b/TODO deleted file mode 100644 index 6630f82f..00000000 --- a/TODO +++ /dev/null @@ -1,30 +0,0 @@ -main installation script and uplaod secrets to kubernetes -kubectl exec -ti -n cozy-linstor srv3 -- vgcreate data /dev/sdb -grafana admin password -grafana redis password -autoconfigure ONCALL_API_URL -oidc -talos setup via tcp-proxy -migrate kubeapps redis to operator -kubeapps patch helm chart to use custom images -flux policies -talos linux firmware -replace reconcile.sh -metallb-configuration -docs: each chart should be self sufficient -docs: core charts must be accessible via helm template | kubectl apply -f -docs: system charts must be accessible via helm install & flux -docs: how to get first token to access cluster -docs: where to store talosconfig -how to version helm charts -autombump chart versions for system charts -move icons to repo -reconcile system helm releases -remove cluster and other namespace resources from apps charts, eg extension-apiserver-authentication-reader -nginx-ingress has no values -update all applications to be managed by operators -fullnameOverride kamaji-etcd -automatically delete provisioned services with the cluster -README Who is the platform intended for? -README how to use platfom -README logo and sreenshots diff --git a/img/cozystack-logo.svg b/img/cozystack-logo.svg new file mode 100644 index 00000000..dd04b4ad --- /dev/null +++ b/img/cozystack-logo.svg @@ -0,0 +1,51 @@ + + + + + + + + diff --git a/manifests/cozystack-installer.yaml b/manifests/cozystack-installer.yaml index 93666eb3..3e27d234 100644 --- a/manifests/cozystack-installer.yaml +++ b/manifests/cozystack-installer.yaml @@ -72,14 +72,14 @@ spec: serviceAccountName: cozystack containers: - name: cozystack - image: "ghcr.io/aenix-io/cozystack/installer:v0.0.1@sha256:d198c1131ed0952dba70918f970bb25764547e64d4e12a455e5b7bfe3040d5e6" + image: "ghcr.io/aenix-io/cozystack/installer:v0.0.1@sha256:e88dd9fa65136863ab8daffdd2deee76cdf5b4d7be4b135115ef0002e97d185a" env: - name: KUBERNETES_SERVICE_HOST value: localhost - name: KUBERNETES_SERVICE_PORT value: "7445" - name: darkhttpd - image: "ghcr.io/aenix-io/cozystack/installer:v0.0.1@sha256:d198c1131ed0952dba70918f970bb25764547e64d4e12a455e5b7bfe3040d5e6" + image: "ghcr.io/aenix-io/cozystack/installer:v0.0.1@sha256:e88dd9fa65136863ab8daffdd2deee76cdf5b4d7be4b135115ef0002e97d185a" command: - /usr/bin/darkhttpd - /cozystack/assets diff --git a/packages/apps/tenant/README.md b/packages/apps/tenant/README.md new file mode 100644 index 00000000..d1c4bfed --- /dev/null +++ b/packages/apps/tenant/README.md @@ -0,0 +1,47 @@ +# Tenant + +A tenant is the main unit of security on the platform. The closest analogy would be Linux kernel namespaces. + +Tenants can be created recursively and are subject to the following rules: + +### Higher-level tenants can access lower-level ones. + +Higher-level tenants can view and manage the applications of all their children. + +### Each tenant has its own domain + +By default (unless otherwise specified), it inherits the domain of its parent with a prefix of its name, for example, if the parent had the domain `example.org`, then `tenant-foo` would get the domain `foo.example.org` by default. + +Kubernetes clusters created in this tenant namespace would get domains like: `kubernetes-cluster.foo.example.org` + +Example: +``` +tenant-root (example.org) +└── tenant-foo (foo.example.org) + └── kubernetes-cluster1 (kubernetes-cluster1.foo.example.org) +``` + +### Lower-level tenants can access the cluster services of their parent (provided they do not run their own) + +Thus, you can create `tenant-u1` with a set of services like `etcd`, `ingress`, `monitoring`. And create another tenant namespace `tenant-u2` inside of `tenant-u1`. + +Let's see what will happen when you run Kubernetes and Postgres under `tenant-u2` namesapce. + +Since `tenant-u2` does not have its own cluster services like `etcd`, `ingress`, and `monitoring`, the applications will use the cluster services of the parent tenant. +This in turn means: + +- The Kubernetes cluster data will be stored in etcd for `tenant-u1`. +- Access to the cluster will be through the common ingress of `tenant-u1`. +- Essentially, all metrics will be collected in the monitoring from `tenant-u1`, and only it will have access to them. + + +Example: +``` +tenant-u1 +├── etcd +├── ingress +├── monitoring +└── tenant-u2 + ├── kubernetes-cluster1 + └── postgres-db1 +``` diff --git a/packages/core/installer/images/installer.json b/packages/core/installer/images/installer.json index ad2c155f..2e0a7634 100644 --- a/packages/core/installer/images/installer.json +++ b/packages/core/installer/images/installer.json @@ -1,14 +1,14 @@ { - "containerimage.config.digest": "sha256:fad46f3695123e4675805045d07394722b6afa36a2fb8becc0af63169585d851", + "containerimage.config.digest": "sha256:b49eb4e818bbedb37ba0447d8c42f17f59746e93c7ed854029abf4f5e7840706", "containerimage.descriptor": { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", - "digest": "sha256:d198c1131ed0952dba70918f970bb25764547e64d4e12a455e5b7bfe3040d5e6", + "digest": "sha256:e88dd9fa65136863ab8daffdd2deee76cdf5b4d7be4b135115ef0002e97d185a", "size": 2074, "platform": { "architecture": "amd64", "os": "linux" } }, - "containerimage.digest": "sha256:d198c1131ed0952dba70918f970bb25764547e64d4e12a455e5b7bfe3040d5e6", + "containerimage.digest": "sha256:e88dd9fa65136863ab8daffdd2deee76cdf5b4d7be4b135115ef0002e97d185a", "image.name": "ghcr.io/aenix-io/cozystack/installer:v0.0.1" } \ No newline at end of file diff --git a/teststand/proxmox/haproxy.cfg b/teststand/proxmox/haproxy.cfg deleted file mode 100644 index 3045b9ef..00000000 --- a/teststand/proxmox/haproxy.cfg +++ /dev/null @@ -1,37 +0,0 @@ -defaults - mode tcp - option dontlognull - timeout http-request 10s - timeout queue 20s - timeout connect 5s - timeout client 5m - timeout server 5m - timeout tunnel 5m - timeout http-keep-alive 10s - timeout check 10s - -frontend kubernetes - bind :::6443 v4v6 - mode tcp - default_backend kubernetes - -frontend talos - bind :::50000 v4v6 - mode tcp - default_backend talos - -backend kubernetes - mode tcp - balance leastconn - default-server observe layer4 error-limit 10 on-error mark-down check - server srv0 192.168.100.11:6443 check - server srv1 192.168.100.12:6443 check - server srv2 192.168.100.13:6443 check - -backend talos - mode tcp - balance leastconn - default-server observe layer4 error-limit 10 on-error mark-down check - server srv0 192.168.100.11:50000 check - server srv1 192.168.100.12:50000 check - server srv2 192.168.100.13:50000 check diff --git a/teststand/proxmox/nftables.conf b/teststand/proxmox/nftables.conf deleted file mode 100644 index 9336a294..00000000 --- a/teststand/proxmox/nftables.conf +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/sbin/nft -f - -flush ruleset - -table inet filter { - chain input { - type filter hook input priority 0; policy drop; - - ct state invalid counter drop comment "early drop of invalid packets" - - ct state {established, related} accept comment "accept all connections established, related" - - iif lo accept comment "accept loopback" - - ip saddr 0.0.0.0/0 tcp dport 22 accept comment "accept ssh" - ip saddr 0.0.0.0/0 tcp dport 8006 accept comment "accept proxmox" - ip saddr 0.0.0.0/0 tcp dport 6443 accept comment "accept kubernetes" - ip saddr 0.0.0.0/0 tcp dport 5000 accept comment "accept talos" - - ip saddr 10.0.0.0/8 accept comment "accept from private networks" - ip saddr 192.168.0.0/16 accept comment "accept from private networks" - - include "/tmp/nftables-*.conf" - - ip protocol icmp accept comment "accept all ICMP types" - - #log prefix "Dropped: " flags all drop comment "dropped packets logger" - #log prefix "Rejected: " flags all reject comment "rejected packets logger" - - counter comment "count dropped packets" - } - chain forward { - type filter hook forward priority 0; policy accept; - } - chain output { - type filter hook output priority 0; policy accept; - } -} - -table ip nat { - chain postrouting { - type nat hook postrouting priority srcnat; policy accept; - oifname "enp41s0" ip saddr 10.0.0.0/8 masquerade comment "masquerade lan" - oifname "enp41s0" ip saddr 192.168.0.0/16 masquerade comment "masquerade lan" - } -}