Without this patch the `holos show buildplans` BuildPlan output has
incorrect yaml with the v3 encoder. For example apiversion: v1alpha5
instead of apiVersion.
Show subcommand:
This is large change that accomplishes a number of goals. First, there
was no convenient way to show a build plan without using the debug logs
to indentify the tags to inject, then calling the cue command with the
right incantation to inspect the BuildPlan.
This patch addresses the problem by adding a `holos show buildplans`
command. The command loads the Platform spec from the platform
directory, then iterates over all Components to produce the BuildPlan.
This patch adds labels and annotations to the platform Components
collection in order to select and filter the output.
Result:
```
❯ holos show components --selector app.holos.run/cluster=local --format=yaml | head
kind: BuildPlan
apiversion: v1alpha5
metadata:
name: podinfo
spec:
artifacts:
- artifact: clusters/local/components/podinfo/podinfo.gen.yaml
generators:
- kind: Helm
output: helm.gen.yaml
```
---
Interface refactor:
This refactors the interface between the `holos` Go CLI layer and the
various core schema data structures. We now use a proper Go interface.
Concurrent execution over platform components has been improved to
accept a closure function so we can use the same interface method to
process the components. We use this to show each component and render
each component from different subcommands using the same interface
embedded in the builder.Platform struct.
The embedded interface allows us to easily swap in different versions,
e.g. v1beta1 and eventually v1. The number of interface methods are
quite small. 14 methods across 4 interfaces in holos/interface.go.
---
Remove old versions:
This patch removes support for versions prior to v1alpha5 in an effort
to clean up cruft.
Previously the Helm generator had no support for the --kube-version
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Previously the Helm generator had no support for the --api-versions
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
This patch strips down the v1alpha4 core and author schemas to only with
is absolutely necessary for all holos users. Aspects of platform
configuration applicable to some, even most, but not all users will be
moved into documentation topics organized as a recipe book.
The functionality removed from the v1alpha4 author schemas in v1alpha5
will move into self contained examples documented as topics on the docs
site.
The overall purpose is to have a focused, composeable, maintainable
author schema to help people get started and ideally we can support for
years with making breaking changes.
With this patch the v1alpha5 helm guide test passes. We're not going to
have this guide anymore but it demonstrates we're back to where we were
with v1alpha4.
Without this patch each version of the core and author schemas are
duplicated into each docs version. This is unnecessary and difficult to
maintain now that we have docusaurus versioned docs enabled.
This patch updates the schema generation script to check if the docs
version has been released, and if so write into a markdown file in the
versioned docs folder. If not, the version is written into the next
version folder.
This patch also updates some, but not all, document links to the md or
mdx relative file paths. This is necessary to generate the correct
versioned links.
A nice outcome of this change is that technical docs no longer need to
link to version specific pages. For example, `[Core Schema]:
./api/core.md` will always refer to the correct auto generated docs
associated with the docs version.
The api references are in reverse order and don't have good descriptions
in the index listings. This patch adds front matter to each generated
document to order them correctly and add a nice description.
Cue uses --inject, -t as the flags to set variables for fields tagged
using @tag(var,type=string).
We used --tag, which is different and requires a mental mapping. Let's
use the same flag and also pass it multiple times like they require so
we can copy and paste the command line output from the debug logs into a
cue export command to see what's going on.
This patch deprecates the --cluster-name flag, use --inject
holos_cluster=mycluster instead.
This patch also removes the environment field from the Component core
API, leaving this to the user namespace to define via tags. We don't
want to be too opinionated on how users manage their platform, baking
environment into the schema is a slippery slope toward those kinds of
opinions.
Closes: #276
Previously #Kubernetes was defined in the platform code. This is a
problem because every platform engineer would need to copy and paste
this code.
This patch moves the #Kubernetes helper into the cue.mod directory so it
can be imported and used ergonomically.
Without this patch the --write-to flag can't be controlled from the
PlatformSpec in the CoreAPI. We need to surface this for the ArgoConfig
struct in the AuthorAPI.
That is to say, in v1alpha3 the --write-to flag was previously assumed
to be deploy/ in ArgoConfig using the DeployFiles functionality. We no
longer have DeployFiles in Core API v1alpha4, all artifacts are instead
written relative to the --write-to flag. Still, we need to expose this
flag in the PlatformSpec so users can use something other than the
deploy directory.
Previously helm charts were cached only by name, which is a problem
because the wrong version would be used when previously cached.
This patch caches charts by name and version to ensure changes in the
version results in pulling the new cached version. It is the user's
responsibility to remove old versions.
This patch also ensures only a single go routine can run cacheChart() at
a time across processes. This is necessary when rendering a platform
because multiple processes will run the Helm generator concurrently, for
example when the same chart is used for multiple environments or
customers.
The mkdir system call serves as the locking mechanism, which is robust
and atomic on all commonly used file systems.
Previously the helm generator was not implemented and returned an error.
This patch is a first pass copying the helm method from
internal/render/helm.go
Basic testing performed with a podinfo chart. It works as the previous
versions in v1alpha3 and before works. This patch does not address the
cached version issue in #273
Previously the Artifact collection was processed sequentially. This
patch provides a modest performance improvement, about 16% faster for
our simple 2 artifact use case, by processing each artifact
concurrently.
Platform rendering provides poor user feedback:
```
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
```
We want to see the metadata.name field of each BuildPlan. This patch
injects the build plan name from the platform spec to make the name
available through the end to end platform rendering process.
Result:
```
❯ holos render platform ./platforms/minimal
rendered stage-namespaces for cluster local in 146.078375ms
rendered prod-namespaces for cluster local in 146.544583ms
rendered test-namespaces for cluster local in 147.0535ms
rendered dev-namespaces for cluster local in 147.499166ms
rendered platform in 147.553875ms
```
The manifest field isn't clear.
Much more clear to have generators produce one Output. Transformers
take multiple Inputs and produce one Output.
The final Transformer, or a single Generator, must produce the final
Artifact.
The Inputs and Output naming to produce an Artifact makes clear the
rendering pipeline we're implementing.
This also makes clear that multiple generators must have at least one
transformer to produce the final output artifact. We model a simple
Join transformer for this case, which is what `holos` was implicitly
doing previously.
Component makes much more sense, that's the domain terminology we use.
BuildContext was meant to be re-used elsewhere, but we never did so the
name serves no purpose.
The repeated enabled booleans and file fields are awkward. It's clear
it's three separate things smashed into one.
kustomize isn't really a generator. It's useless because there is no
way to reference a plain file in a component directory.
This patch replaces the kustomize generator with a file generator which
simply reads one single file. Multiple of these generators may be used
to read one or more files.
Then, kustomize may transform these generated files, which are generated
by simply reading from the filesystem.
This API is much improved over the previous.
```
kind: BuildPlan
apiVersion: v1alpha4
metadata:
name: prod-namespaces
spec:
component: projects/platform/components/namespaces
steps:
- artifact: clusters/no-cluster/components/prod-namespaces/prod-namespaces.gen.yaml
generators:
- kind: Resources
manifest: resources.gen.yaml
resources:
Namespace:
prod-jeff:
metadata:
name: prod-jeff
labels:
kubernetes.io/metadata.name: prod-jeff
kind: Namespace
apiVersion: v1
prod-gary:
metadata:
name: prod-gary
labels:
kubernetes.io/metadata.name: prod-gary
kind: Namespace
apiVersion: v1
prod-nate:
metadata:
name: prod-nate
labels:
kubernetes.io/metadata.name: prod-nate
kind: Namespace
apiVersion: v1
transformers:
- kind: Kustomize
kustomize:
kustomization:
commonLabels:
holos.run/component.name: prod-namespaces
resources:
- resources.gen.yaml
- application.gen.yaml
- artifact: clusters/no-cluster/gitops/prod-namespaces.gen.yaml
generators:
- kind: Resources
manifest: application.gen.yaml
resources:
Application:
argocd:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prod-namespaces
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
project: default
source:
path: examples/v1alpha4/deploy/clusters/no-cluster/components/prod-namespaces
repoURL: https://github.com/holos-run/bank-of-holos
targetRevision: main
transformers:
- kind: Kustomize
kustomize:
kustomization:
commonLabels:
holos.run/component.name: prod-namespaces
resources:
- resources.gen.yaml
- application.gen.yaml
```
A build step either produces kubernetes objects or a gitops manifest.
Both are effectively the same, they're just kubernetes resources.
For the use case of applying common labels to both, we'll have the
Author API pass the same Kustomization to two separate build steps. One
step to produce the resources, a second to produce the argocd
application or flux kustomization.
Each step produces a manifest and a gitops file, so we need a unique
name for each step. The most common case will be a single build step
matching the name of the build plan itself.
The kustomize transformer needs a filename to store the output from
generators so it has an input for the transformer. This patch adds
fields for each kind of generator so the kustomize.#Kustomization can be
configured with the files `holos` will write generated output to.
This patch implements the v1alpha4 component rendering builder for a
component BuildPlan. We don't yet have the CUE definitions, so this
hasn't been end to end tested yet, the next step is defining the
generators and transforms in the core API BuildPlan.
The holos cli does not use an interface to handle different Platform api
versions. This makes it difficult to evolve the API in a backwards
compatible way.
This patch adds a top level switch statement to the `holos render
platform` command. The switch discriminates on the Platform API
version. v1alpha3 and earlier are classified as legacy versions and
will use the existing strict types. v1alpha4 and later versions will
use an interface to render the platform, allowing for multiple types to
implement the platform rendering interface.
Previously the CUE code needed to specify the Platform.spec.model field,
which created friction. This patch adds a cue struct tag to unify the
field with an open struct.
❯ holos render platform ./platform --log-level=debug
could not run: could not marshal cue instance platform: cue: marshal error: spec.model: cannot convert incomplete value "_" to JSON at internal/builder/platform.go:45
spec.model: cannot convert incomplete value "_" to JSON
The render command completes successfully with this patch without the
user having to provide a value for the spec.model field.
The API docs are not published yet becuase the module is private. Our
own docs site does not have any API reference docs.
This patch adds auto-generated markdown docs for the core v1alpha2 types
by generating them directly from the go source code.
Some light editing of the output of `gomarkdoc` is necessary to get the
heading anchor tags to align correctly for Docusaurus.
Previously a couple of methods were defined on the Result struct.
This patch moves the methods to an internal wrapper struct to remove
them from the API documentation.
With this patch the API between holos and CUE is entirely a data API.
The pod identity webhook component fails to render with v1alpha2. This
patch fixes the problem by providing concrete values for enableHooks and
the namespace of the helm chart holos component.
The namespace is mainly necessary to render the ArgoCD Application
resource along side the helm chart output.
With this patch the eso-creds-manager component renders correctly. This
is a `#Kubernetes` type build plan which uses the
spec.components.resources map to manage resources.
The only issue was needing to provide the namespace to the nested holos
component inside the BuildPlan.
The ArgoCD Application resource moves to the DeployFiles field of a
separate holos component in the same build plan at
spec.components.resources.argocd. For this reason a separate Result
object is no longer necessary inside of the Holos cli for the purpose of
managing Flux or ArgoCD gitops. The CUE code can simply inline whatever
gitops resources it wants and the holos cli will write the files
relative to the cluster specific deploy directory.
Result:
```
❯ holos render component --cluster-name management components/eso-creds-manager
2:55PM INF result.go:195 wrote deploy file version=0.84.1 path=deploy/clusters/management/gitops/eso-creds-manager.application.gen.yaml bytes=350
2:55PM INF render.go:92 rendered eso-creds-manager version=0.84.1 cluster=management name=eso-creds-manager status=ok action=rendered
```
Previously methods were defined on the API objects in the v1alpha1 API.
The API should be data structures only. This patch refactors the
methods responsible for orchestrating the build plan to pull them into
the internal render package.
The result is the API is cleaner and has no methods. The render package
has corresponding data structures which simply wrap around the API
structure and implement the methods to render and return the result to
the CLI.
This commit compiles, but it has not been tested at all. It's almost
surely broken completely.
Previously in v1alpha1, all Holos structs are located in the same
package. This makes it difficult to focus on only the structs necessary
to transfer configuration data from CUE to the `holos` cli.
This patch splits the structs into `meta` and `core` where the core
package holds the structs end users should refer to and focus on. Only
the Platform resource is in core now, but other BuildPlan types will be
added shortly.
