To properly support the kubectl-slice use case the Artifact map needs to
write multiple files out to the filesystem. This needs to be dynamic in
the sense holos and the end user don't know what files the kubectl-slice
transformer is producing.
As a hack, which may actually turn out to be "good enough" this patch
makes the Slice transformer behave like so:
1. Execute kubectl-slice outputting to an empty temp directory.
2. Holos saves all files in this directory into the artifact map.
3. At the end of the Artifact pipeline, if the final artifact produced
ends in a /, then all keys in the artifact map having the prefix are
written to the output directory.
This should be sufficient for the use case, but we'll need to consider
how this transformer and apporach works when subsequent transformers are
used in the pipeline. I haven't thought deeply about it, but it should
ideally work pretty well if the tools involved truly only care about
directories and not the files within the directory.
Copied from kustomize, spike for spiarh.
Result: head to artifact.go next.
could not run: could not save deploy/slice/components/slice: open deploy/slice/components/slice: is a directory at internal/artifact/artifact.go:71
could not run: could not render component: could not run command:
holos '--log-level' 'debug' '--log-format' 'console' 'render' 'component' '--inject' 'outputBaseDir=slice' '--inject' 'holos_component_name=slice' '--inject' 'holos_component_path=components/slice' '--inject' 'holos_component_labels={"holos.run/component.name":"slice"}' '--inject' 'holos_component_annotations={"app.holos.run/description":"slice transformer"}' './components/slice'
exit status 1 at cli/render/render.go:171
Relevant debug logs:
running command: kubectl 'kustomize' '/var/folders/22/zt67pphj6h1fgknqfy23ppl80000gn/T/holos.kustomize4273326823'
tmp: removed
running command: kubectl-slice '-f' '/var/folders/22/zt67pphj6h1fgknqfy23ppl80000gn/T/holos.slice2683550041/slice.gen.yaml' '-o' '/var/folders/22/zt67pphj6h1fgknqfy23ppl8000
0gn/T/holos.slice2683550041/slice'
storing: /var/folders/22/zt67pphj6h1fgknqfy23ppl80000gn/T/holos.slice2683550041/slice/deployment-httpbin.yaml
storing: /var/folders/22/zt67pphj6h1fgknqfy23ppl80000gn/T/holos.slice2683550041/slice/service-httpbin.yaml
tmp: removed
Without this patch migrating from [helm hierarchies] to Holos requires
the user to unify the value hierarchy. This is a problem because helm
hierarchies are difficult to unify because it's not clear if or why a
value is used in the final results. This makes it difficult to identify
how to resolve conflicts.
This patch adds `valueFiles` field to the Helm component kind. This
field is intended to provide a direct migration path from the
ApplicationSet.spec.template.spec.sources.helm.valueFiles field. With
this patch, users can directly migrate the values files to CUE using
`@embed`, then directly migrate the valueFiles field to reference the
values from within CUE.
Note we actively discourage the use of Helm value hierarchies. The
feature is intended as a temporary migration tool. We encourage the use
of CUE unification instead. After migration, the valueFiles field
should be refactored to the values field as one unified structure in
CUE. The valueFiles field makes this second order migration easier
becuase we can inspect and verify the complete rendered output, allowing
us to determine if a value is actually used in the final configuration
or is overridden.
[helm hierarchies]: https://medium.com/containers-101/using-helm-hierarchies-in-multi-source-argo-cd-applications-for-promoting-to-different-gitops-133c3bc93678
Without this patch selectors don't work as expected. This patch
fixes selectors such that each --selector flag value configures one
selector containing multiple positive or negative label matchers.
Result:
Render build plans for cluster dev or cluster test. Note the use of two
flags indicating logical OR.
holos render platform --selector cluster=test --selector cluster=dev
rendered external-secrets for cluster test in 299.897542ms
rendered external-secrets for cluster dev in 299.9225ms
rendered external-secrets-crds for cluster test in 667.6075ms
rendered external-secrets-crds for cluster dev in 708.126541ms
rendered platform in 708.795625ms
Render build plans for prod clusters that are not customer facing. Note
the use of one selector with comma separated labels.
holos render platform --selector "tier=prod,scope!=customer"
mpvl suggests @embed is a more ideal solution than our implementation of
core.Component.Instances for the use case of unifying YAML data updated
by Kargo Stage resources.
See the issue for a link to the discussion.
Previously holos render platform was not setting the --extract-yaml file
when calling holos render component, causing data file instances defined
in the Platform spec to be discarded.
This patch passes the value along using the flag.
Extract YAML is more clear and aligns with the schema docs for the
Component Instance field which has an extractYAML kind. This also
leaves the door open for additional kinds of data extractors which are
almost certainly going to be needed.
Previously there isn't a good way to unify json and yaml files with the
cue configuration. This is a problem for use cases where data can be
generated idempotentialy prior to rendering the platform configuration.
The first use case is to explore unifying configuration with decrypted
sops values, which isn't typical since Holos is designed to handle
secrets with ExternalSecret resources, but does fit into the use case of
executing a command to produce data idempotently, then make the data
available to the platform configuration.
Other use cases this feature is intended to support are the prior
experiment where we fetch top level platform configuration from an rpc
service, and the future goal of integrating with data provided by
Terraform.
Previously holos unconditionally executed helm repo add which failed for
private repositories requiring basic authentication.
This patch addresses the problem by using the Helm SDK to pull and cache
charts without adding them as repositories. New fields for the
core.Helm type allow basic auth credentials to be read from environment
variables.
Multiple repositories are supported by using different env vars for
different repositories.
Without this patch we do not support installing Kargo from an OCI helm
chart. We want to support:
```
Component: #Helm & {
Name: "kargo"
Namespace: Kargo.Namespace
Chart: {
name: "oci://ghcr.io/akuity/kargo-charts/kargo"
version: "1.0.3"
release: Name
}
EnableHooks: true
Values: Kargo.Values
}
```
This patch fixes the problem by using the base name for filesystem cache
operations.
Previously, build tags were not propagated from `holos render platform
-t validate` through to the underlying `holos render component` command.
This is a problem because validators need to be selectively enabled as a
work around until we have an audit mode field.
This patch fixes the problem by propagating command line tags from the
render platform command to the underlying commands. This patch also
propagates tags for the show command.
Previously the BuildPlan pipeline didn't execute generators and
transformers concurrently. All steps were sequentially executed. Holos
was primarily concurrent by executing multiple BuildPlans at once.
This patch changes the Build implementation for each BuildPlan to
execute a GoRoutine pipeline. One producer fans out to a group of
routines each executing the pipeline for one artifact in the build plan.
The pipeline has 3 stages:
1: Fan-out to build each Generator concurrently.
2: Fan-in to build each Transformer sequentially.
3: Fan-out again to run each validator concurrently.
When the artifact pipelines return, the producer closes the tasks
channel causing the worker tasks to return.
Note the overall runtime for 8 BuildPlans is roughly equivalent to
previously at 160ms with --concurrency=8 on my M3 Max. I expect this to
perform better than previously when multiple artifacts are rendered for
each BuildPlan.
Without this patch the BuildPlan resulting from a Platform that has
components with labels and annotations does not have the labels or
annotations of the source component.
Holos should copy the labels and annotations defined on each of the
Platform.spec.components to the resulting BuildPlan so end users can see
clearly where a BuildPlan originated from, and filter with selectors the
intermediate output BuildPlan the same way we filter with selectors the
original Platform spec components list.
Result:
```
holos init platform v1alpha5 --force
holos show buildplans | head
```
```yaml
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: podinfo
labels:
app.holos.run/cluster: local
app.holos.run/name: podinfo
annotations:
app.holos.run/description: podinfo for cluster local
```
Without this patch the holos show buildplans command returns results in
an inconsistent order. This is a problem because the output should be
idempotent.
This patch fixes the problem by adding an EncodeSeq(idx int, v any) method to
the encoder interface. idx represents the index position of the
Platform.spec.components list after selector filtering has been applied.
This patch modifies the json and yaml encoders to buffer out of order
results from the concurrent go routines.
Result:
Concurrent execution is preserved. The buffer is kept to a reasonable
size, entries are deleted once they're encoded in the correct order.
Most importantly the output is consistent and idempotent so we can write
effective integration tests.
Sometimes, but not always, the holos show buildplans command produces no
output.
```
❯ holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug
finalized config from flags
rendered platform in 13.458µs
```
It only happens when there's a selector. It doesn't happen without the
selector flag. It only happens with ==, not with =.
This test fails quickly.
```
while [[ $(holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug | wc -l) -eq 39 ]]; do true; done
```
This test runs until killed.
```
while [[ $(holos show buildplans --log-level=debug | wc -l) -eq 279 ]]; do true; done
```
Solution:
The problem is the use of the map. Iterating over the keys happens in a
random order. With the fix we check in an explicit order.
Show subcommand:
This is large change that accomplishes a number of goals. First, there
was no convenient way to show a build plan without using the debug logs
to indentify the tags to inject, then calling the cue command with the
right incantation to inspect the BuildPlan.
This patch addresses the problem by adding a `holos show buildplans`
command. The command loads the Platform spec from the platform
directory, then iterates over all Components to produce the BuildPlan.
This patch adds labels and annotations to the platform Components
collection in order to select and filter the output.
Result:
```
❯ holos show components --selector app.holos.run/cluster=local --format=yaml | head
kind: BuildPlan
apiversion: v1alpha5
metadata:
name: podinfo
spec:
artifacts:
- artifact: clusters/local/components/podinfo/podinfo.gen.yaml
generators:
- kind: Helm
output: helm.gen.yaml
```
---
Interface refactor:
This refactors the interface between the `holos` Go CLI layer and the
various core schema data structures. We now use a proper Go interface.
Concurrent execution over platform components has been improved to
accept a closure function so we can use the same interface method to
process the components. We use this to show each component and render
each component from different subcommands using the same interface
embedded in the builder.Platform struct.
The embedded interface allows us to easily swap in different versions,
e.g. v1beta1 and eventually v1. The number of interface methods are
quite small. 14 methods across 4 interfaces in holos/interface.go.
---
Remove old versions:
This patch removes support for versions prior to v1alpha5 in an effort
to clean up cruft.
Previously the Helm generator had no support for the --kube-version
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Previously the Helm generator had no support for the --api-versions
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Previously the holos command line expected a Platform and BuildPlan
resource at the top level of the exported data from CUE. This forced us
to use hidden fields for everything else.
This patch modifies the BuildData struct to first look for a holos top
level field and use it if present. This opens up other top level fields
for use by end users.
Our intent is to reserve any top level field prefixed with holos.
Note this follows how Timoni works as well.
This patch strips down the v1alpha4 core and author schemas to only with
is absolutely necessary for all holos users. Aspects of platform
configuration applicable to some, even most, but not all users will be
moved into documentation topics organized as a recipe book.
The functionality removed from the v1alpha4 author schemas in v1alpha5
will move into self contained examples documented as topics on the docs
site.
The overall purpose is to have a focused, composeable, maintainable
author schema to help people get started and ideally we can support for
years with making breaking changes.
With this patch the v1alpha5 helm guide test passes. We're not going to
have this guide anymore but it demonstrates we're back to where we were
with v1alpha4.
Cue uses --inject, -t as the flags to set variables for fields tagged
using @tag(var,type=string).
We used --tag, which is different and requires a mental mapping. Let's
use the same flag and also pass it multiple times like they require so
we can copy and paste the command line output from the debug logs into a
cue export command to see what's going on.
This patch deprecates the --cluster-name flag, use --inject
holos_cluster=mycluster instead.
This patch also removes the environment field from the Component core
API, leaving this to the user namespace to define via tags. We don't
want to be too opinionated on how users manage their platform, baking
environment into the schema is a slippery slope toward those kinds of
opinions.
Closes: #276
Without this patch holos render platform may hang until the overall
context timeout is reached. This is a problem because the user has no
idea why it's hung.
This patch adds a warning at the 5 second and another at the 10 second
mark indicating the lock may be deadlocked. The user can then remove
the directory.
Without this patch kustomize errors aren't surfaced when executing holos
render platform.
This patch gives a fighting chance to the user to figure out what's
going on. The stderr is copied, logged, and surfaced up to the parent
holos render platform command.
Previously the #Helm and #Kustomize build plan helpers were not defined
in the v1alpha4 Author API. We need this definition to update the
Quickstart guide for v1alpha4 from v1alpha3.
This patch defines the #Helm and #Kustomize helpers in the Author API
similar to how #Kubernetes is defined.
Without this patch the --write-to flag can't be controlled from the
PlatformSpec in the CoreAPI. We need to surface this for the ArgoConfig
struct in the AuthorAPI.
That is to say, in v1alpha3 the --write-to flag was previously assumed
to be deploy/ in ArgoConfig using the DeployFiles functionality. We no
longer have DeployFiles in Core API v1alpha4, all artifacts are instead
written relative to the --write-to flag. Still, we need to expose this
flag in the PlatformSpec so users can use something other than the
deploy directory.
Previously the file generator was unimplemented. This patch implements
it as a simple file read into the ArtifactMap for use by the Kustomize
or Join transformers.
With this patch all v1alpha4 Core API features are implemented.
Resources, Helm, and File generators. Kustomize and Join transformers.
Blank lines show up in the output which is confusing. This patch fixes
the only source location identified with the following command.
export HOLOS_LOG_LEVEL=debug
export HOLOS_LOG_FORMAT=json
holos render platform ./platform 2>&1 | jq -r 'select (.msg == "")'
Previously helm charts were cached only by name, which is a problem
because the wrong version would be used when previously cached.
This patch caches charts by name and version to ensure changes in the
version results in pulling the new cached version. It is the user's
responsibility to remove old versions.
This patch also ensures only a single go routine can run cacheChart() at
a time across processes. This is necessary when rendering a platform
because multiple processes will run the Helm generator concurrently, for
example when the same chart is used for multiple environments or
customers.
The mkdir system call serves as the locking mechanism, which is robust
and atomic on all commonly used file systems.
Previously the helm generator was not implemented and returned an error.
This patch is a first pass copying the helm method from
internal/render/helm.go
Basic testing performed with a podinfo chart. It works as the previous
versions in v1alpha3 and before works. This patch does not address the
cached version issue in #273
holos.FilePath is intended for paths relative to the platform root
directory. We use the Artifact to store lots of stuff not related to
the platform root directory, for example kustomization.yaml in a temp
dir. Most entries are not relative to the platform root directory given
the implicit cfg.WriteTo prefix.
Previously:
could not run: could not build dev-join: could not get foo.yaml: not set at builder/v1alpha4/builder.go:180
This is confusing because set has nothing to do with the missing input
from the cue code the user writes.
Result:
could not run: could not build test-join: missing foo.yaml at builder/v1alpha4/builder.go:180
This is better because it at doesn't distract the user from the fact
they're missing a foo.yaml generator output to align with the
transformer input.
The code was inlined in a number of places, it makes sense to move it to
the interface. It'll also make it easier to test, we can provide a null
writer concrete value.
Previously the Artifact collection was processed sequentially. This
patch provides a modest performance improvement, about 16% faster for
our simple 2 artifact use case, by processing each artifact
concurrently.
Platform rendering provides poor user feedback:
```
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
```
We want to see the metadata.name field of each BuildPlan. This patch
injects the build plan name from the platform spec to make the name
available through the end to end platform rendering process.
Result:
```
❯ holos render platform ./platforms/minimal
rendered stage-namespaces for cluster local in 146.078375ms
rendered prod-namespaces for cluster local in 146.544583ms
rendered test-namespaces for cluster local in 147.0535ms
rendered dev-namespaces for cluster local in 147.499166ms
rendered platform in 147.553875ms
```
With this patch the first use case of CUE Resources + Kustomize is fully
working, artifacts are written into the deploy directory.
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
The output indicates we need to plumb the BuildPlan metadata.name from
the PlatfromSpec through to the render component command. This is
necessary so we can report the correct name instead of just the base
path.
Without this patch holos writes a single yaml document that is a list.
It needs to write a file that contains multiple documents, each document
a map[string]any representing the kubernetes resource.
This patch fixes the problem. With this patch kustomize fully executes.
