Previously it wasn't clear for users if platform wide structs should be
definitions or hidden fields in CUE. They should be hidden fields when
they contain data and definitions when they define a schema.
This patch updates the generate platform v1alpha4 subcommand to use the
correct field names consistently for clarity.
The api references are in reverse order and don't have good descriptions
in the index listings. This patch adds front matter to each generated
document to order them correctly and add a nice description.
Without this patch it's difficult to mix in a plain file as a config
map. This is necessary for the use case of using a Job to generate a
secret in-cluster. We want a plain shell script to be carried through
and transformed into the job.
We already have the KustomizeConfig fields to support this, they just
weren't wired up to the #Kustomization component kind.
I didn't check if it's wired up to Helm and Kustomize for expedience.
They may be missing there as well.
Cue uses --inject, -t as the flags to set variables for fields tagged
using @tag(var,type=string).
We used --tag, which is different and requires a mental mapping. Let's
use the same flag and also pass it multiple times like they require so
we can copy and paste the command line output from the debug logs into a
cue export command to see what's going on.
This patch deprecates the --cluster-name flag, use --inject
holos_cluster=mycluster instead.
This patch also removes the environment field from the Component core
API, leaving this to the user namespace to define via tags. We don't
want to be too opinionated on how users manage their platform, baking
environment into the schema is a slippery slope toward those kinds of
opinions.
Closes: #276
Now that we have CommonLabels as part of the ComponentConfig for all
components, it makes sense to also mix in CommonLabels for a Project.
Common labes are key aspect of the Technical Overview document.
For the Author API, it would be nice to define a schema for the fields
common to all component kinds. Users could then configure all kinds by
unifying the schema into their own platform tree.
This makes a clear use case to extract the common fields back into an
embedded struct like we did in v1alpha3. I removed the embedded struct
in v1alpha4 because it wasn't clear why it should be separate, but now
the use case is clear, to configure all component kinds.
Without this patch holos render platform may hang until the overall
context timeout is reached. This is a problem because the user has no
idea why it's hung.
This patch adds a warning at the 5 second and another at the 10 second
mark indicating the lock may be deadlocked. The user can then remove
the directory.
The Kustomize build plan kind needs to support both copying files from
the component directory and pulling resources from https URL's. Without
this patch this support is missing from the Author API
With this patch the Kustomize build plan kind has a KustomizeConfig
field with two structs, Files and Resources. The kustomization
resources list is built up from both of these.
Two transformers are used so we don't affect the GitOps transfomer which
really only needs CommonLabels.
I decided to keep this field exclusive to the Kustomize kind, but it
could replace the Kustomization field of the other kinds as well.
Without this patch the user facing API doesn't have a way to kustomize
the output of all the build plan kinds. This patch ensures the
Kustomization field is present on all of Helm, Kustomize, and
Kubernetes.
This field is inteded for patches and transforms. The second
kustomization in the transformer sequence is intended for common labels
and annotations, managed by a corresponding field instead of a full on
Kustomization resource.
Fix:
could not run: could not marshal json projects/platform/components/cert-manager: cue: marshal error: spec.artifacts.0.generators.0.helm.enableHooks: cannot convert incomplete value "bool" to JSON at internal/builder/builder.go:63
spec.artifacts.0.generators.0.helm.enableHooks: cannot convert incomplete value "bool" to JSON:
/Users/jeff/Holos/bank-of-holos/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha4/types_go_gen.cue:235:16
could not run: could not render component: exit status 1 at builder/v1alpha4/builder.go:94
Without this patch kustomize errors aren't surfaced when executing holos
render platform.
This patch gives a fighting chance to the user to figure out what's
going on. The stderr is copied, logged, and surfaced up to the parent
holos render platform command.
Previously the #Helm and #Kustomize build plan helpers were not defined
in the v1alpha4 Author API. We need this definition to update the
Quickstart guide for v1alpha4 from v1alpha3.
This patch defines the #Helm and #Kustomize helpers in the Author API
similar to how #Kubernetes is defined.
Previously #Kubernetes was defined in the platform code. This is a
problem because every platform engineer would need to copy and paste
this code.
This patch moves the #Kubernetes helper into the cue.mod directory so it
can be imported and used ergonomically.
This patch gets the Author API rendering the namespaces component in the
Bank of Holos guide. It's not the final form of the API yet, we still
need to decide how best to expose the Kubernetes, Helm, and Kustomize
definitions.
I'm thinking we abstract away the transformers and generators within the
Author API Kubernetes definition.
Without this patch the --write-to flag can't be controlled from the
PlatformSpec in the CoreAPI. We need to surface this for the ArgoConfig
struct in the AuthorAPI.
That is to say, in v1alpha3 the --write-to flag was previously assumed
to be deploy/ in ArgoConfig using the DeployFiles functionality. We no
longer have DeployFiles in Core API v1alpha4, all artifacts are instead
written relative to the --write-to flag. Still, we need to expose this
flag in the PlatformSpec so users can use something other than the
deploy directory.
Previously the file generator was unimplemented. This patch implements
it as a simple file read into the ArtifactMap for use by the Kustomize
or Join transformers.
With this patch all v1alpha4 Core API features are implemented.
Resources, Helm, and File generators. Kustomize and Join transformers.
Blank lines show up in the output which is confusing. This patch fixes
the only source location identified with the following command.
export HOLOS_LOG_LEVEL=debug
export HOLOS_LOG_FORMAT=json
holos render platform ./platform 2>&1 | jq -r 'select (.msg == "")'
Previously helm charts were cached only by name, which is a problem
because the wrong version would be used when previously cached.
This patch caches charts by name and version to ensure changes in the
version results in pulling the new cached version. It is the user's
responsibility to remove old versions.
This patch also ensures only a single go routine can run cacheChart() at
a time across processes. This is necessary when rendering a platform
because multiple processes will run the Helm generator concurrently, for
example when the same chart is used for multiple environments or
customers.
The mkdir system call serves as the locking mechanism, which is robust
and atomic on all commonly used file systems.
Previously the helm generator was not implemented and returned an error.
This patch is a first pass copying the helm method from
internal/render/helm.go
Basic testing performed with a podinfo chart. It works as the previous
versions in v1alpha3 and before works. This patch does not address the
cached version issue in #273
holos.FilePath is intended for paths relative to the platform root
directory. We use the Artifact to store lots of stuff not related to
the platform root directory, for example kustomization.yaml in a temp
dir. Most entries are not relative to the platform root directory given
the implicit cfg.WriteTo prefix.
Previously:
could not run: could not build dev-join: could not get foo.yaml: not set at builder/v1alpha4/builder.go:180
This is confusing because set has nothing to do with the missing input
from the cue code the user writes.
Result:
could not run: could not build test-join: missing foo.yaml at builder/v1alpha4/builder.go:180
This is better because it at doesn't distract the user from the fact
they're missing a foo.yaml generator output to align with the
transformer input.
The code was inlined in a number of places, it makes sense to move it to
the interface. It'll also make it easier to test, we can provide a null
writer concrete value.
Previously the Artifact collection was processed sequentially. This
patch provides a modest performance improvement, about 16% faster for
our simple 2 artifact use case, by processing each artifact
concurrently.
Platform rendering provides poor user feedback:
```
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
```
We want to see the metadata.name field of each BuildPlan. This patch
injects the build plan name from the platform spec to make the name
available through the end to end platform rendering process.
Result:
```
❯ holos render platform ./platforms/minimal
rendered stage-namespaces for cluster local in 146.078375ms
rendered prod-namespaces for cluster local in 146.544583ms
rendered test-namespaces for cluster local in 147.0535ms
rendered dev-namespaces for cluster local in 147.499166ms
rendered platform in 147.553875ms
```
With this patch the first use case of CUE Resources + Kustomize is fully
working, artifacts are written into the deploy directory.
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
The output indicates we need to plumb the BuildPlan metadata.name from
the PlatfromSpec through to the render component command. This is
necessary so we can report the correct name instead of just the base
path.
Without this patch holos writes a single yaml document that is a list.
It needs to write a file that contains multiple documents, each document
a map[string]any representing the kubernetes resource.
This patch fixes the problem. With this patch kustomize fully executes.
