Provisioner cluster:
This patch creates a Certificate resource in the provisioner for each
host associated with the project. By default, one host is created for
each stage with the short hostname set to the project name.
A namespace is also created for each project for eso creds refresher to
manage service accounts for SecretStore resources in the workload
clusters.
Workload cluster:
For each env, plus one system namespace per stage:
- Namespace per env
- SecretStore per env
- ExternalSecret per host in the env
Common names for the holos project, prod stage:
- holos.k1.ois.run
- holos.k2.ois.run
- holos.ois.run
Common names for the holos project, dev stage:
- holos.dev.k1.ois.run
- holos.dev.k2.ois.run
- holos.dev.ois.run
- holos.gary.k1.ois.run
- holos.gary.k2.ois.run
- holos.gary.ois.run
- holos.jeff.k1.ois.run
- holos.jeff.k2.ois.run
- holos.jeff.ois.run
- holos.nate.k1.ois.run
- holos.nate.k2.ois.run
- holos.nate.ois.run
Usage:
holos render --cluster-name=provisioner \
~/workspace/holos-run/holos/docs/examples/platforms/reference/clusters/provisioner/projects/...
holos render --cluster-name=k1 \
~/workspace/holos-run/holos/docs/examples/platforms/reference/clusters/workload/projects/...
holos render --cluster-name=k2 \
~/workspace/holos-run/holos/docs/examples/platforms/reference/clusters/workload/projects/...
This patch introduces a new BuildPlan spec.components.resources
collection, which is a map version of
spec.components.kubernetesObjectsList. The map version is much easier
to work with and produce in CUE than the list version.
The list version should be deprecated and removed prior to public
release.
The projects holos instance renders multiple holos components, each
containing kubernetes api objects defined directly in CUE.
<project>-system is intended for the ext auth proxy providers for all
stages.
<project>-namespaces is intended to create a namespace for each
environment in the project.
The intent is to expand the platform level definition of a project to
include the per-stage auth proxy and per-env role bindings. Secret
Store and ESO creds refresher resources will also be defined by the
platform level definition of a project.
This patch establishes the BuildPlan struct as the single API contract
between CUE and Holos. A BuildPlan spec contains a list of each of the
support holos component types.
The purpose of this data structure is to support the use case of one CUE
instance generating 1 build plan that contains 0..N of each type of
holos component.
The need for multiple components per one CUE instance is to support the
generation of a collection of N~4 flux kustomization resources per
project and P~6 projects built from one CUE instance.
Tested with:
holos render --cluster-name=k2 ~/workspace/holos-run/holos/docs/examples/platforms/reference/clusters/foundation/cloud/init/namespaces/...
Common labels are removed because they're too tightly coupled to the
model of one component per one cue instance.
This patch refactors the go structs used to decode cue output for
processing by the holos cli. For context, the purpose of the structs
are to inform holos how the data from cue should be modeled and
processed as a rendering pipeline that provides rendered yaml to
configure kubernetes api objects.
The structs share common fields in the form of the HolosComponent
embedded struct. The three main holos component kinds today are:
1. KubernetesObjects - CUE outputs a nested map where each value is a
single rendered api object (resource).
2. HelmChart - CUE outputs the chart name and values. Holos calls helm
template to render the chart. Additional api objects may be
overlaid into the rendered output. Kustomize may also optionally be
called at the end of the render pipeline.
3. KustomizeBuild - CUE outputs data to construct a kustomize
kustomization build. The holos component contains raw yaml files to
use as kustomization resources. CUE optionally defines additional
patches, common labels, etc.
With the Go structs, cue may directly import the definitions to more
easily keep the CUE definitions in sync with what the holos cli expects
to receive.
The holos component types may be imported into cue using:
cue get go github.com/holos-run/holos/api/v1alpha1/...
Without this patch ks/prod-iam-zitadel often gets blocked waiting for
jobs that will never complete. In addition, flux should not manage the
zitadel-test-connection Pod which is an unnecessary artifact of the
upstream helm chart.
We'd disable helm hooks, but they're necessary to create the init and
setup jobs.
This patch also changes the default behavior of Kustomizations from
wait: true to wait: false. Waiting is expensive for the api server and
slows down the reconciliation process considerably.
Component authors should use ks.spec.healthChecks to target specific
important resources to watch and wait for.
This patch fixes the problem of the actions runner scale set listener
pod failing every 3 seconds. See
https://github.com/actions/actions-runner-controller/issues/3351
The solution is not ideal, if the primary cluster is down workflows will
not execute. The primary cluster shouldn't go down though so this is
the trade off. Lower log spam and resource usage by eliminating the
failing pods on other clusters for lower availability if the primary
cluster is not available.
We could let the pods loop and if the primary is unavailable another
would quickly pick up the role, but it doesn't seem worth it.
The effect of this patch is limited to refreshing credentials only for
namespaces that exist in the local cluster. There is structure in place
in the CUE code to allow for namespaces bound to specific clusters, but
this is used only by the optional Vault component.
This patch was an attempt to work around
https://github.com/actions/actions-runner-controller/issues/3351 by
deploying the runner scale sets into unique namespaces.
This effort was a waste of time, only one listener pod successfully
registered for a given scale set name / group combination.
Because we have only one group named Default we can only have one
listener pod globally for a given scale set name.
Because we want our workflows to execute regardless of the availability
of a single cluster, we're going to let this fail for now. The pod
retries every 3 seconds. When a cluster is destroyed, another cluster
will quickly register.
A follow up patch will look to expand this retry behavior.
This patch migrates the vault component from [holos-infra][1] to a cue
based component. Vault is optional in the reference platform, so this
patch also defines an `#OptionalServices` struct to conditionally manage
a service across multiple clusters in the platform.
The primary use case for optional services is managing a namespace to
provision and provide secrets across clusters.
[1]: https://github.com/holos-run/holos-infra/tree/v0.5.0/components/core/core/vault
Pods are unnecessarily created when deploying helm based holos
components and often fail. Prevent these test pods by disabling helm
hooks with the `--no-hooks` flag.
Closes: #54
Problem:
The standby cluster on k2 fails to start. A pgbackrest pod first
restores the database from S3, then the pgha nodes try to replay the WAL
as part of the standby initialization process. This fails because the
PGDATA directory is not empty.
Solution:
Specify the spec.dataSource field only when the cluster is configured as
a primary cluster.
Result:
Non-primary clusters are standby, they skip the pgbackrest job to
restore from S3 and move straight to patroni replaying the WAL from S3
as part of the pgha pods.
One of the two pgha pods becomes the "standby leader" and restores the
WAL from S3. The other is a cascading standby and then restores the
same WAL from the standby leader.
After 8 minutes both pods are ready.
```
❯ k get pods
NAME READY STATUS RESTARTS AGE
zitadel-pgbouncer-d9f8cffc-j469g 2/2 Running 0 11m
zitadel-pgbouncer-d9f8cffc-xq29g 2/2 Running 0 11m
zitadel-pgha1-27w7-0 4/4 Running 0 11m
zitadel-pgha1-c5qj-0 4/4 Running 0 11m
zitadel-repo-host-0 2/2 Running 0 11m
```
Problem:
The k3 and k4 clusters are getting the Zitadel components which are
really only intended for the core cluster pair.
Solution:
Split the workload subtree into two, named foundation and accounts. The
core cluster pair gets foundation+accounts while the kX clusters get
just the foundation subtree.
Result:
prod-zitadel-iam is no longer managed on k3 and k4
Set the restore point to time="2024-03-11T17:08:58Z" level=info
msg="crunchy-pgbackrest ends" which is just after Gary and Nate
registered and were granted the cluster-admin role.
The [Streaming Standby][standby] architecture requires custom tls certs
for two clusters in two regions to connect to each other.
This patch manages the custom certs following the configuration
described in the article [Using Cert Manager to Deploy TLS for Postgres
on Kubernetes][article].
NOTE: One thing not mentioned anywhere in the crunchy documentation is
how custom tls certs work with pgbouncer. The pgbouncer service uses a
tls certificate issued by the pgo root cert, not by the custom
certificate authority.
For this reason, we use kustomize to patch the zitadel Deployment and
the zitadel-init and zitadel-setup Jobs. The patch projects the ca
bundle from the `zitadel-pgbouncer` secret into the zitadel pods at
/pgbouncer/ca.crt
[standby]: https://access.crunchydata.com/documentation/postgres-operator/latest/architecture/disaster-recovery#streaming-standby-with-an-external-repo
[article]: https://www.crunchydata.com/blog/using-cert-manager-to-deploy-tls-for-postgres-on-kubernetes
A full backup was taken using:
```
kubectl annotate postgrescluster zitadel postgres-operator.crunchydata.com/pgbackrest-backup="$(date)"
```
And completed with:
```
❯ k logs -f zitadel-backup-5r6v-v5jnm
time="2024-03-10T21:52:15Z" level=info msg="crunchy-pgbackrest starts"
time="2024-03-10T21:52:15Z" level=info msg="debug flag set to false"
time="2024-03-10T21:52:15Z" level=info msg="backrest backup command requested"
time="2024-03-10T21:52:15Z" level=info msg="command to execute is [pgbackrest backup --stanza=db --repo=2 --type=full]"
time="2024-03-10T21:55:18Z" level=info msg="crunchy-pgbackrest ends"
```
This patch verifies the point in time backup is robust in the face of
the following operations:
1. pg cluster zitadel was deleted (whole namespace emptied)
2. pg cluster zitadel was re-created _without_ a `dataSource`
3. pgo initailized a new database and backed up the blank database to
S3.
4. pg cluster zitadel was deleted again.
5. pg cluster zitadel was re-created with `dataSource` `options: ["--type=time", "--target=\"2024-03-10 21:56:00+00\""]` (Just after the full backup completed)
6. Restore completed successfully.
7. Applied the holos zitadel component.
8. Zitadel came up successfully and user login worked as expected.
- [x] Perform an in place [restore][restore] from [s3][bucket].
- [x] Set repo1-retention-full to clear warning
[restore]: https://access.crunchydata.com/documentation/postgres-operator/latest/tutorials/backups-disaster-recovery/disaster-recovery#restore-properties
[bucket]: https://access.crunchydata.com/documentation/postgres-operator/latest/tutorials/backups-disaster-recovery/disaster-recovery#cloud-based-data-source
To establish the canonical https://login.ois.run identity issuer on the
core cluster pair.
Custom resources for PGO have been imported with:
timoni mod vendor crds -f deploy/clusters/core2/components/prod-pgo-crds/prod-pgo-crds.gen.yaml
Note, the zitadel tls connection took some considerable effort to get
working. We intentionally use pgo issued certs to reduce the toil of
managing certs issued by cert manager.
The default tls configuration of pgo is pretty good with verify full
enabled.
The core2 cluster cannot provision pvcs because it's using the k8s-dev
pool when it has credentials valid only for the k8s-prod pool.
This patch adds an entry to the platform cluster map to configure the
pool for each cluster, with a default of k8s-dev.
PGO uses plain yaml and kustomize as the recommended installation
method. Holos supports upstream by adding a new PlainFiles component
kind, which simply copies files into place and lets kustomize handle the
generation of the api objects.
Cue is responsible for very little in this kind of component, basically
allowing overlay resources if needed and deferring everything else to
the holos cli.
The holos cli in turn is responsible for executing kubectl kustomize
build on the input directory to produce the rendered output, then writes
the rendered output into place.
Without this patch the arc controller fails to create a listener. The
template for the listener doesn't appear to be configurable from the
chart.
Could patch the listener pod template with kustomize, do this as a
follow up feature.
With this patch we get the expected two pods in the runner system
namespace:
```
❯ k get pods
NAME READY STATUS RESTARTS AGE
gha-rs-7db9c9f7-listener 1/1 Running 0 43s
gha-rs-controller-56bb9c77d9-6tjch 1/1 Running 0 8s
```
The resource names for the arc controller are too long:
❯ k get pods -n arc-systems
NAME READY STATUS RESTARTS AGE
gha-runner-scale-set-controller-gha-rs-controller-6bdf45bd6jx5n 1/1 Running 0 59m
Solve the problem by allowing components to set the release name to
`gha-rs-controller` which requires an additional field from the cue code
to differentiate from the chart name.
Separate the SecretStore resources from the namespaces component because
it creates a deadlock. The secretstore crds don't get applied until the
eso component is managed.
The namespaces component should have nothing but core api objects, no
custom resources.
This patch switches CockroachDB to use certs provided by ExternalSecrets
instead of managing Certificate resources in-cluster from the upstream
helm chart.
This paves the way for multi-cluster replication by moving certificates
outside of the lifecycle of the workload cluster cockroach db operates
within.
Closes: #36
Issuing mtls certs for cockroach db moves to the provisioner cluster so
we can more easily support cross cluster replication in the future.
crdb certs will be synced same as public tls certs, using ExternalSecret
resources.
This patch uses cert manager in the provisioner cluster to provision tls
certs for https://login.example.com and https://httpbin.k2.example.com
The certs are not yet synced to the clusters. Next step is to replace
the Certificate resources with ExternalSecret resources, then remove
cert manager from the workload clusters.
This patch moves certificate management to the provisioner cluster to
centralize all secrets into the highly secured cluster. This change
also simplifies the architecture in a number of ways:
1. Certificate lives are now completely independent of cluster
lifecycle.
2. Remove the need for bi-directional sync to save cert secrets.
3. Workload clusters no longer need access to DNS.
Multiple holos components rely on kustomize to modify the output of the
upstream helm chart, for example patching a Deployment to inject the
istio sidecar.
The new holos cue based component system did not support running
kustomize after helm template. This patch adds the kustomize execution
if two fields are defined in the helm chart kind of cue output.
The API spec is pretty loose in this patch but I'm proceeding for
expedience and to inform the final API with more use cases as more
components are migrated to cue.
