The api references are in reverse order and don't have good descriptions
in the index listings. This patch adds front matter to each generated
document to order them correctly and add a nice description.
Cue uses --inject, -t as the flags to set variables for fields tagged
using @tag(var,type=string).
We used --tag, which is different and requires a mental mapping. Let's
use the same flag and also pass it multiple times like they require so
we can copy and paste the command line output from the debug logs into a
cue export command to see what's going on.
This patch deprecates the --cluster-name flag, use --inject
holos_cluster=mycluster instead.
This patch also removes the environment field from the Component core
API, leaving this to the user namespace to define via tags. We don't
want to be too opinionated on how users manage their platform, baking
environment into the schema is a slippery slope toward those kinds of
opinions.
Closes: #276
Now that we have CommonLabels as part of the ComponentConfig for all
components, it makes sense to also mix in CommonLabels for a Project.
Common labes are key aspect of the Technical Overview document.
For the Author API, it would be nice to define a schema for the fields
common to all component kinds. Users could then configure all kinds by
unifying the schema into their own platform tree.
This makes a clear use case to extract the common fields back into an
embedded struct like we did in v1alpha3. I removed the embedded struct
in v1alpha4 because it wasn't clear why it should be separate, but now
the use case is clear, to configure all component kinds.
The Kustomize build plan kind needs to support both copying files from
the component directory and pulling resources from https URL's. Without
this patch this support is missing from the Author API
With this patch the Kustomize build plan kind has a KustomizeConfig
field with two structs, Files and Resources. The kustomization
resources list is built up from both of these.
Two transformers are used so we don't affect the GitOps transfomer which
really only needs CommonLabels.
I decided to keep this field exclusive to the Kustomize kind, but it
could replace the Kustomization field of the other kinds as well.
Without this patch the user facing API doesn't have a way to kustomize
the output of all the build plan kinds. This patch ensures the
Kustomization field is present on all of Helm, Kustomize, and
Kubernetes.
This field is inteded for patches and transforms. The second
kustomization in the transformer sequence is intended for common labels
and annotations, managed by a corresponding field instead of a full on
Kustomization resource.
Previously the #Helm and #Kustomize build plan helpers were not defined
in the v1alpha4 Author API. We need this definition to update the
Quickstart guide for v1alpha4 from v1alpha3.
This patch defines the #Helm and #Kustomize helpers in the Author API
similar to how #Kubernetes is defined.
Previously #Kubernetes was defined in the platform code. This is a
problem because every platform engineer would need to copy and paste
this code.
This patch moves the #Kubernetes helper into the cue.mod directory so it
can be imported and used ergonomically.
This patch gets the Author API rendering the namespaces component in the
Bank of Holos guide. It's not the final form of the API yet, we still
need to decide how best to expose the Kubernetes, Helm, and Kustomize
definitions.
I'm thinking we abstract away the transformers and generators within the
Author API Kubernetes definition.
Previously helm charts were cached only by name, which is a problem
because the wrong version would be used when previously cached.
This patch caches charts by name and version to ensure changes in the
version results in pulling the new cached version. It is the user's
responsibility to remove old versions.
This patch also ensures only a single go routine can run cacheChart() at
a time across processes. This is necessary when rendering a platform
because multiple processes will run the Helm generator concurrently, for
example when the same chart is used for multiple environments or
customers.
The mkdir system call serves as the locking mechanism, which is robust
and atomic on all commonly used file systems.
Previously the helm generator was not implemented and returned an error.
This patch is a first pass copying the helm method from
internal/render/helm.go
Basic testing performed with a podinfo chart. It works as the previous
versions in v1alpha3 and before works. This patch does not address the
cached version issue in #273
Previously the Artifact collection was processed sequentially. This
patch provides a modest performance improvement, about 16% faster for
our simple 2 artifact use case, by processing each artifact
concurrently.
Platform rendering provides poor user feedback:
```
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
```
We want to see the metadata.name field of each BuildPlan. This patch
injects the build plan name from the platform spec to make the name
available through the end to end platform rendering process.
Result:
```
❯ holos render platform ./platforms/minimal
rendered stage-namespaces for cluster local in 146.078375ms
rendered prod-namespaces for cluster local in 146.544583ms
rendered test-namespaces for cluster local in 147.0535ms
rendered dev-namespaces for cluster local in 147.499166ms
rendered platform in 147.553875ms
```
The manifest field isn't clear.
Much more clear to have generators produce one Output. Transformers
take multiple Inputs and produce one Output.
The final Transformer, or a single Generator, must produce the final
Artifact.
The Inputs and Output naming to produce an Artifact makes clear the
rendering pipeline we're implementing.
This also makes clear that multiple generators must have at least one
transformer to produce the final output artifact. We model a simple
Join transformer for this case, which is what `holos` was implicitly
doing previously.
Component makes much more sense, that's the domain terminology we use.
BuildContext was meant to be re-used elsewhere, but we never did so the
name serves no purpose.
The repeated enabled booleans and file fields are awkward. It's clear
it's three separate things smashed into one.
kustomize isn't really a generator. It's useless because there is no
way to reference a plain file in a component directory.
This patch replaces the kustomize generator with a file generator which
simply reads one single file. Multiple of these generators may be used
to read one or more files.
Then, kustomize may transform these generated files, which are generated
by simply reading from the filesystem.
This API is much improved over the previous.
```
kind: BuildPlan
apiVersion: v1alpha4
metadata:
name: prod-namespaces
spec:
component: projects/platform/components/namespaces
steps:
- artifact: clusters/no-cluster/components/prod-namespaces/prod-namespaces.gen.yaml
generators:
- kind: Resources
manifest: resources.gen.yaml
resources:
Namespace:
prod-jeff:
metadata:
name: prod-jeff
labels:
kubernetes.io/metadata.name: prod-jeff
kind: Namespace
apiVersion: v1
prod-gary:
metadata:
name: prod-gary
labels:
kubernetes.io/metadata.name: prod-gary
kind: Namespace
apiVersion: v1
prod-nate:
metadata:
name: prod-nate
labels:
kubernetes.io/metadata.name: prod-nate
kind: Namespace
apiVersion: v1
transformers:
- kind: Kustomize
kustomize:
kustomization:
commonLabels:
holos.run/component.name: prod-namespaces
resources:
- resources.gen.yaml
- application.gen.yaml
- artifact: clusters/no-cluster/gitops/prod-namespaces.gen.yaml
generators:
- kind: Resources
manifest: application.gen.yaml
resources:
Application:
argocd:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prod-namespaces
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
project: default
source:
path: examples/v1alpha4/deploy/clusters/no-cluster/components/prod-namespaces
repoURL: https://github.com/holos-run/bank-of-holos
targetRevision: main
transformers:
- kind: Kustomize
kustomize:
kustomization:
commonLabels:
holos.run/component.name: prod-namespaces
resources:
- resources.gen.yaml
- application.gen.yaml
```
A build step either produces kubernetes objects or a gitops manifest.
Both are effectively the same, they're just kubernetes resources.
For the use case of applying common labels to both, we'll have the
Author API pass the same Kustomization to two separate build steps. One
step to produce the resources, a second to produce the argocd
application or flux kustomization.
Each step produces a manifest and a gitops file, so we need a unique
name for each step. The most common case will be a single build step
matching the name of the build plan itself.
The kustomize transformer needs a filename to store the output from
generators so it has an input for the transformer. This patch adds
fields for each kind of generator so the kustomize.#Kustomization can be
configured with the files `holos` will write generated output to.
This patch implements the v1alpha4 component rendering builder for a
component BuildPlan. We don't yet have the CUE definitions, so this
hasn't been end to end tested yet, the next step is defining the
generators and transforms in the core API BuildPlan.
The holos cli does not use an interface to handle different Platform api
versions. This makes it difficult to evolve the API in a backwards
compatible way.
This patch adds a top level switch statement to the `holos render
platform` command. The switch discriminates on the Platform API
version. v1alpha3 and earlier are classified as legacy versions and
will use the existing strict types. v1alpha4 and later versions will
use an interface to render the platform, allowing for multiple types to
implement the platform rendering interface.
Previously all generated ArgoCD Application resources go into the
default project following the Quickstart guide. The configuration code
is being organized into the concept of projects in the filesystem, so we
want to the GitOps configuration to also reflect this concept of
projects.
This patch extends the ArgoConfig user facing schema to accept a project
string. The app-projects component automatically manages AppProject
resources in the argocd namespace for each of the defined projects.
This allows CUE configuration in the a project directory to specify the
project name so that all Applications are automatically assigned to the
correct project.
Previously, the #Resources struct listing valid resources to use with
APIObjects in each of the components types was closed. This made it
very difficult for users to mix in new resources and use the Kubernetes
component kind.
This patch moves the definition of the valid resources to package holos
from the schema API. The schema still enforces some light constraints,
but doesn't keep the struct closed.
A new convention is introduced in the form of configuring all components
using _ComponentConfig defined at the root, then unifying this struct
with all of the component kinds. See schema.gen.cue for how this works.
This approach enables mixing in ArgoCD applications to all component
kinds, not just Helm as was done previously. Similarly, the
user-constrained #Resources definition unifies with all component kinds.
It's OK to leave the yaml.Marshall in the schema API. The user
shouldn't ever have to deal with #APIObjects, instead they should pass
Resources through the schema API which will use APIObjects to create
apiObjectMap for each component type and the BuildPlan.
This is still more awkward than I want, but it's a good step in the
right direction.
This patch adds the httpbin routes component. It's missing the
Certificate component, the next step is to wire up automatic certificate
management in the gateway configuration, which is a prime use case for
holos. Similar to how we register components and namespaces, we'll
register certificates.
This patch also adds the #Platform.Domain field to the user facing
schema API. We previously stored the domain in the Model but it makes
sense to lift it up to the Platform and have a sensible default value
for it.
Another example of #237 needing to be addressed soon.
This patch adds the schema api for the Kubernetes build plan, which
produces plain API resources directly from CUE. It's needed for the
namespaces component which is foundational to many of our guides.
The first guide that needs this is the expose a service guide, we need
to register the namespaces from the istio component.
The Expose a Service doc is meant to be the second step after the
Quickstart doc. This commit adds the section describing how to install
the Gateway API.
The Kustomize build plan is introduced at this point in a similar way
the Helm build plan was introduced in the quickstart.
Having the management cluster hard coded into the definition of the
standard fleets is problematic for guides that don't need a management
cluster.
Define the fleets, but leave the set of clusters empty until they're
needed.
Previously, each document needed to be manually included in the sidebars
to show up. In addition, index paths like /docs/ and /docs/guides/ were
not found.
This patch addresses both problems by switching sidebars to
automatically generate from filesystem directories. Important documents
like the getting started guide and introduction are expected to add a
`slug: /foo` front matter item to create a permalink.
The result is the sidebar reflects the filesystem while the URL bar is
more of a permalink. Files should be able to be moved around the file
system and the sidebar tree without affecting their URL.
This patch also consolidates the API and Docs sidebars into one.
Previously, the quickstart step of generating the pod info component and
generating the platform as a whole left the task of integrating the
Component into the Platform as an exercise for the reader. This is a
problem because it creates unnecessary friction.
This patch addresses the problem by lifting up the Platform concept
into the user-facing Schema API. The generated platform includes a top
level #Platform definition which exposes the core Platform specification
on the Output field.
The Platform CUE instance then reduces to a simple `#Platform.Output`
which provides the Platform spec to holos for rendering each component
for each cluster.
The CUE code for the schema.#Platform iterates over each
Component to derive the list of components to manage for the Platform.
The CUE code for the generated quickstart platform links the definition
of StandardFleets, which is a Workload fleet and a Management cluster
fleet to the Platform conveninece wrapper.
Finally, the generated podinfo component drops a CUE file at the
repository root to automatically add the component to every workload
cluster.
The result is the only task left for the end user is to define at least
one workload cluster. Once defined, the component is automatically
managed because it is managed on all workload clusters.
This approach futher opens the door to allow generated components to
define their namespaces and generated secrets on the management cluster
separate from their workloads on the workload clusters.
This patch includes a behavior change, from now on all generated
components should assume they are writing to the root of the user's Git
repository so that they can generate files through the whole tree.
In the future, we should template output paths for generated components.
A simple approach might be to embed a file with a .target suffix, with
the contents being a simple Go template of the file path to write to.
The holos generate subcommand can then check if any given embedded file
foo has a foo.target companion, then write the target to the rendered
template value.
Previosly the end user needed to write, or at least copy and paste, a
large amount of boiler plate code to achieve the goal of declaring a
helm chart component. There is a gap between the cue code:
(#Helm & Chart).Output
And the full BuildPlan produced for the Holos cli to execute the
rendering process. The boiler plate code in schema.cue at the root of
the platform infrastructure repository was largely responsible for
defining how a BuildPlan with one HelmChart component is derived from
this #Helm definition.
This patch moves the definitions into a new, documented API named
`schema`. End users are expected to define their own #Helm definition
using the schema.#Helm, like so in the root level schema.cue:
#Helm: schema.#Helm
