This package clarifies and simplifies the input protocol of the holos
compile command. A BuildPlanRequest represents the complete context
necessary to compile a BuildPlan.
This patch also upgrades cue to v0.13.0-alpha4
This patch changes the behavior of all commands to execute in the
current working directory of the platform root cue module. This gets
cue vet working well, otherwise it complains about fully qualified
paths. The cue command line expects the current working directory to be
a cue module.
The Env field of the Command schema wasn't implemented, so we remove it
from the schema. It's unclear it's necessary. Setting the environment
in the parent context should suffice for current use cases.
Name it consistently, Context can be a bit ambiguous. Also define the
tags used to pass data from the platform layer to the component layer as
go constants in the core api for clear documentation.
Previously, holos loaded the full CUE Instance to discriminate against
the APIVersion and Kind. This is a problem because we need to know the
api version to know what tags to inject. For example, v1alpha6 needs to
have the build context injected but v1alpha5 does not.
This patch fixes the problem by changing holos render component to look
for a `typemeta.yaml` file in the component directory. The command line
tool discriminates on the kind and apiVersion specified in this file
prior to building the CUE instance.
Result:
We're able to inject version specific cue tags into the CUE instance.
Note previously the build context was passed to CUE by writing a file to
the filesystem and using the embed functionality. We cannot proceed
with this embed approach because the filesystem writes are not safe for
concurrent use. Therefore, we inject the build context as a tag which
is safe for concurrent use.
Previously, the build plan CUE code has no way to refer to the temporary
directory managed and owned by the `holod render component` command.
This patch adds a placeholder context field for holos to fill in just
before exporting the final build plan from CUE.
This allows the user to refer to the tempDir field without the value
being concrete, useful for constructing an argument vector for the new
Command task.
Without this patch the default apiVersion for the Platform resource is
v1alpha5 even though we're using the v1alpha6 schemas. This patch
updates the default value and re-generates the docs and cue schema using
`make generate`
Before:
holos show platform
apiVersion: v1alpha5
kind: Platform
metadata:
name: default
spec:
components: []
Result:
holos init platform v1alpha6 --force
holos show platform
apiVersion: v1alpha6
kind: Platform
metadata:
name: default
spec:
components: []
Rendering the platform fails as expected:
holos render platform
could not run: unsupported version: v1alpha6 at internal/builder/platform.go:102
Without this patch the v1alpha6 of the author schema imports v1alpha5 of
the core schema. This patch imports the v1alpha6 core schema.
This is a necessary step to get holos render platform working after
holos init platform v1alpha6.
Result: We still have other things to copy over to establish v1alpha6.
holos render platform
could not run: holos.metadata.name: cannot convert non-concrete value string at builder/v1alpha5/builder.go:34
holos.metadata.name: cannot convert non-concrete value string:
/Users/jeff/Holos/command/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha6/types_go_gen.cue:305:2
Without this patch migrating from [helm hierarchies] to Holos requires
the user to unify the value hierarchy. This is a problem because helm
hierarchies are difficult to unify because it's not clear if or why a
value is used in the final results. This makes it difficult to identify
how to resolve conflicts.
This patch adds `valueFiles` field to the Helm component kind. This
field is intended to provide a direct migration path from the
ApplicationSet.spec.template.spec.sources.helm.valueFiles field. With
this patch, users can directly migrate the values files to CUE using
`@embed`, then directly migrate the valueFiles field to reference the
values from within CUE.
Note we actively discourage the use of Helm value hierarchies. The
feature is intended as a temporary migration tool. We encourage the use
of CUE unification instead. After migration, the valueFiles field
should be refactored to the values field as one unified structure in
CUE. The valueFiles field makes this second order migration easier
becuase we can inspect and verify the complete rendered output, allowing
us to determine if a value is actually used in the final configuration
or is overridden.
[helm hierarchies]: https://medium.com/containers-101/using-helm-hierarchies-in-multi-source-argo-cd-applications-for-promoting-to-different-gitops-133c3bc93678
Previously holos render platform was not setting the --extract-yaml file
when calling holos render component, causing data file instances defined
in the Platform spec to be discarded.
This patch passes the value along using the flag.
Extract YAML is more clear and aligns with the schema docs for the
Component Instance field which has an extractYAML kind. This also
leaves the door open for additional kinds of data extractors which are
almost certainly going to be needed.
Previously there isn't a good way to unify json and yaml files with the
cue configuration. This is a problem for use cases where data can be
generated idempotentialy prior to rendering the platform configuration.
The first use case is to explore unifying configuration with decrypted
sops values, which isn't typical since Holos is designed to handle
secrets with ExternalSecret resources, but does fit into the use case of
executing a command to produce data idempotently, then make the data
available to the platform configuration.
Other use cases this feature is intended to support are the prior
experiment where we fetch top level platform configuration from an rpc
service, and the future goal of integrating with data provided by
Terraform.
Previously holos unconditionally executed helm repo add which failed for
private repositories requiring basic authentication.
This patch addresses the problem by using the Helm SDK to pull and cache
charts without adding them as repositories. New fields for the
core.Helm type allow basic auth credentials to be read from environment
variables.
Multiple repositories are supported by using different env vars for
different repositories.
Without this patch the BuildPlan resulting from a Platform that has
components with labels and annotations does not have the labels or
annotations of the source component.
Holos should copy the labels and annotations defined on each of the
Platform.spec.components to the resulting BuildPlan so end users can see
clearly where a BuildPlan originated from, and filter with selectors the
intermediate output BuildPlan the same way we filter with selectors the
original Platform spec components list.
Result:
```
holos init platform v1alpha5 --force
holos show buildplans | head
```
```yaml
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: podinfo
labels:
app.holos.run/cluster: local
app.holos.run/name: podinfo
annotations:
app.holos.run/description: podinfo for cluster local
```
Sometimes, but not always, the holos show buildplans command produces no
output.
```
❯ holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug
finalized config from flags
rendered platform in 13.458µs
```
It only happens when there's a selector. It doesn't happen without the
selector flag. It only happens with ==, not with =.
This test fails quickly.
```
while [[ $(holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug | wc -l) -eq 39 ]]; do true; done
```
This test runs until killed.
```
while [[ $(holos show buildplans --log-level=debug | wc -l) -eq 279 ]]; do true; done
```
Solution:
The problem is the use of the map. Iterating over the keys happens in a
random order. With the fix we check in an explicit order.
Without this patch the `holos show buildplans` BuildPlan output has
incorrect yaml with the v3 encoder. For example apiversion: v1alpha5
instead of apiVersion.
Show subcommand:
This is large change that accomplishes a number of goals. First, there
was no convenient way to show a build plan without using the debug logs
to indentify the tags to inject, then calling the cue command with the
right incantation to inspect the BuildPlan.
This patch addresses the problem by adding a `holos show buildplans`
command. The command loads the Platform spec from the platform
directory, then iterates over all Components to produce the BuildPlan.
This patch adds labels and annotations to the platform Components
collection in order to select and filter the output.
Result:
```
❯ holos show components --selector app.holos.run/cluster=local --format=yaml | head
kind: BuildPlan
apiversion: v1alpha5
metadata:
name: podinfo
spec:
artifacts:
- artifact: clusters/local/components/podinfo/podinfo.gen.yaml
generators:
- kind: Helm
output: helm.gen.yaml
```
---
Interface refactor:
This refactors the interface between the `holos` Go CLI layer and the
various core schema data structures. We now use a proper Go interface.
Concurrent execution over platform components has been improved to
accept a closure function so we can use the same interface method to
process the components. We use this to show each component and render
each component from different subcommands using the same interface
embedded in the builder.Platform struct.
The embedded interface allows us to easily swap in different versions,
e.g. v1beta1 and eventually v1. The number of interface methods are
quite small. 14 methods across 4 interfaces in holos/interface.go.
---
Remove old versions:
This patch removes support for versions prior to v1alpha5 in an effort
to clean up cruft.
Previously the Helm generator had no support for the --kube-version
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Previously the Helm generator had no support for the --api-versions
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
This patch strips down the v1alpha4 core and author schemas to only with
is absolutely necessary for all holos users. Aspects of platform
configuration applicable to some, even most, but not all users will be
moved into documentation topics organized as a recipe book.
The functionality removed from the v1alpha4 author schemas in v1alpha5
will move into self contained examples documented as topics on the docs
site.
The overall purpose is to have a focused, composeable, maintainable
author schema to help people get started and ideally we can support for
years with making breaking changes.
With this patch the v1alpha5 helm guide test passes. We're not going to
have this guide anymore but it demonstrates we're back to where we were
with v1alpha4.
Without this patch each version of the core and author schemas are
duplicated into each docs version. This is unnecessary and difficult to
maintain now that we have docusaurus versioned docs enabled.
This patch updates the schema generation script to check if the docs
version has been released, and if so write into a markdown file in the
versioned docs folder. If not, the version is written into the next
version folder.
This patch also updates some, but not all, document links to the md or
mdx relative file paths. This is necessary to generate the correct
versioned links.
A nice outcome of this change is that technical docs no longer need to
link to version specific pages. For example, `[Core Schema]:
./api/core.md` will always refer to the correct auto generated docs
associated with the docs version.
The api references are in reverse order and don't have good descriptions
in the index listings. This patch adds front matter to each generated
document to order them correctly and add a nice description.
Cue uses --inject, -t as the flags to set variables for fields tagged
using @tag(var,type=string).
We used --tag, which is different and requires a mental mapping. Let's
use the same flag and also pass it multiple times like they require so
we can copy and paste the command line output from the debug logs into a
cue export command to see what's going on.
This patch deprecates the --cluster-name flag, use --inject
holos_cluster=mycluster instead.
This patch also removes the environment field from the Component core
API, leaving this to the user namespace to define via tags. We don't
want to be too opinionated on how users manage their platform, baking
environment into the schema is a slippery slope toward those kinds of
opinions.
Closes: #276
Now that we have CommonLabels as part of the ComponentConfig for all
components, it makes sense to also mix in CommonLabels for a Project.
Common labes are key aspect of the Technical Overview document.
For the Author API, it would be nice to define a schema for the fields
common to all component kinds. Users could then configure all kinds by
unifying the schema into their own platform tree.
This makes a clear use case to extract the common fields back into an
embedded struct like we did in v1alpha3. I removed the embedded struct
in v1alpha4 because it wasn't clear why it should be separate, but now
the use case is clear, to configure all component kinds.
The Kustomize build plan kind needs to support both copying files from
the component directory and pulling resources from https URL's. Without
this patch this support is missing from the Author API
With this patch the Kustomize build plan kind has a KustomizeConfig
field with two structs, Files and Resources. The kustomization
resources list is built up from both of these.
Two transformers are used so we don't affect the GitOps transfomer which
really only needs CommonLabels.
I decided to keep this field exclusive to the Kustomize kind, but it
could replace the Kustomization field of the other kinds as well.
Without this patch the user facing API doesn't have a way to kustomize
the output of all the build plan kinds. This patch ensures the
Kustomization field is present on all of Helm, Kustomize, and
Kubernetes.
This field is inteded for patches and transforms. The second
kustomization in the transformer sequence is intended for common labels
and annotations, managed by a corresponding field instead of a full on
Kustomization resource.
Previously the #Helm and #Kustomize build plan helpers were not defined
in the v1alpha4 Author API. We need this definition to update the
Quickstart guide for v1alpha4 from v1alpha3.
This patch defines the #Helm and #Kustomize helpers in the Author API
similar to how #Kubernetes is defined.
Previously #Kubernetes was defined in the platform code. This is a
problem because every platform engineer would need to copy and paste
this code.
This patch moves the #Kubernetes helper into the cue.mod directory so it
can be imported and used ergonomically.
This patch gets the Author API rendering the namespaces component in the
Bank of Holos guide. It's not the final form of the API yet, we still
need to decide how best to expose the Kubernetes, Helm, and Kustomize
definitions.
I'm thinking we abstract away the transformers and generators within the
Author API Kubernetes definition.
Without this patch the --write-to flag can't be controlled from the
PlatformSpec in the CoreAPI. We need to surface this for the ArgoConfig
struct in the AuthorAPI.
That is to say, in v1alpha3 the --write-to flag was previously assumed
to be deploy/ in ArgoConfig using the DeployFiles functionality. We no
longer have DeployFiles in Core API v1alpha4, all artifacts are instead
written relative to the --write-to flag. Still, we need to expose this
flag in the PlatformSpec so users can use something other than the
deploy directory.
Previously helm charts were cached only by name, which is a problem
because the wrong version would be used when previously cached.
This patch caches charts by name and version to ensure changes in the
version results in pulling the new cached version. It is the user's
responsibility to remove old versions.
This patch also ensures only a single go routine can run cacheChart() at
a time across processes. This is necessary when rendering a platform
because multiple processes will run the Helm generator concurrently, for
example when the same chart is used for multiple environments or
customers.
The mkdir system call serves as the locking mechanism, which is robust
and atomic on all commonly used file systems.
Previously the helm generator was not implemented and returned an error.
This patch is a first pass copying the helm method from
internal/render/helm.go
Basic testing performed with a podinfo chart. It works as the previous
versions in v1alpha3 and before works. This patch does not address the
cached version issue in #273
