Previously the Command core type was only useful for Validators and was
a bit hacky in that Holos appends the directory to the end of the
argument vector.
This patch changes the Command type to represent a generic command for
use as a Generator, Transformer, or Validator. The type is extended to
support the semantics of reading output from a file, directory, or
standard output pipe.
Go templates are used to fill in Holos managed data values, for example
the temporary directory associated with the pipeline task. The TaskData
structu represents these values passed into the template engine.
To properly support the kubectl-slice use case the Artifact map needs to
write multiple files out to the filesystem. This needs to be dynamic in
the sense holos and the end user don't know what files the kubectl-slice
transformer is producing.
As a hack, which may actually turn out to be "good enough" this patch
makes the Slice transformer behave like so:
1. Execute kubectl-slice outputting to an empty temp directory.
2. Holos saves all files in this directory into the artifact map.
3. At the end of the Artifact pipeline, if the final artifact produced
ends in a /, then all keys in the artifact map having the prefix are
written to the output directory.
This should be sufficient for the use case, but we'll need to consider
how this transformer and apporach works when subsequent transformers are
used in the pipeline. I haven't thought deeply about it, but it should
ideally work pretty well if the tools involved truly only care about
directories and not the files within the directory.
Copied from kustomize, spike for spiarh.
Result: head to artifact.go next.
could not run: could not save deploy/slice/components/slice: open deploy/slice/components/slice: is a directory at internal/artifact/artifact.go:71
could not run: could not render component: could not run command:
holos '--log-level' 'debug' '--log-format' 'console' 'render' 'component' '--inject' 'outputBaseDir=slice' '--inject' 'holos_component_name=slice' '--inject' 'holos_component_path=components/slice' '--inject' 'holos_component_labels={"holos.run/component.name":"slice"}' '--inject' 'holos_component_annotations={"app.holos.run/description":"slice transformer"}' './components/slice'
exit status 1 at cli/render/render.go:171
Relevant debug logs:
running command: kubectl 'kustomize' '/var/folders/22/zt67pphj6h1fgknqfy23ppl80000gn/T/holos.kustomize4273326823'
tmp: removed
running command: kubectl-slice '-f' '/var/folders/22/zt67pphj6h1fgknqfy23ppl80000gn/T/holos.slice2683550041/slice.gen.yaml' '-o' '/var/folders/22/zt67pphj6h1fgknqfy23ppl8000
0gn/T/holos.slice2683550041/slice'
storing: /var/folders/22/zt67pphj6h1fgknqfy23ppl80000gn/T/holos.slice2683550041/slice/deployment-httpbin.yaml
storing: /var/folders/22/zt67pphj6h1fgknqfy23ppl80000gn/T/holos.slice2683550041/slice/service-httpbin.yaml
tmp: removed
Without this patch there are unexpected lint errors in version 1.60
where 1.61.0 passes locally on my machine.
This patch updates to:
golangci-lint has version 1.64.5 built with go1.24.0 from 0a603e49 on 2025-02-13T21:19:55Z
Previously using make bump to bump a version did not also update all of
the test cases and documentation to reflect the new version. This patch
updates the make bump tasks call HOLOS_UPDATE_SCRIPTS=1 scripts/test to
keep the test cases and documentation in sync with the new version.
Previously the tests fail because they were not updated to use the new
version string in holos, or the new topo sort behavior in cue 0.12.0.
This patch updates the test cases using:
HOLOS_UPDATE_SCRIPTS=1 scripts/test
Result: make test passes
PROBLEM:
The "Kustomize" tutorial has hardcoded code blocks and hasn't been
updated to use the automated testscript workflow.
SOLUTION:
Create a test for the Kustomize tutorial.
Create a testscript for the Kustomize test.
Update the Kustomize MDX file to load in data from the testscript directory.
OUTCOME:
The code content in the Kustomize tutorial now comes directly from the
testscript workflow.
PROBLEM:
The "Hello Holos" tutorial has hardcoded code blocks and hasn't been
updated to use the automated testscript workflow.
SOLUTION:
* Create a test for the Hello Holos tutorial.
* Create a testscript for the Hello Holos test.
* Update the Hello Holos MDX file to load in data from the testscript directory.
OUTCOME:
The code content in the Hello Holos tutorial now comes directly from the
testscript workflow.
* Convert all files with.period.separators to hyphen-separators.
* Rename and markdown_test.go to be specific to Helm Values.
* Move helm-values_test.go to be in the same directory as the Helm Values doc.
* Move Blackbox common configuration CUE file to `config/prometheus` so it can be imported as necessary.
* Use explicit import statements for Blackbox common config in `blackbox` and `prometheus` components.
Closes: #399
Without this patch migrating from [helm hierarchies] to Holos requires
the user to unify the value hierarchy. This is a problem because helm
hierarchies are difficult to unify because it's not clear if or why a
value is used in the final results. This makes it difficult to identify
how to resolve conflicts.
This patch adds `valueFiles` field to the Helm component kind. This
field is intended to provide a direct migration path from the
ApplicationSet.spec.template.spec.sources.helm.valueFiles field. With
this patch, users can directly migrate the values files to CUE using
`@embed`, then directly migrate the valueFiles field to reference the
values from within CUE.
Note we actively discourage the use of Helm value hierarchies. The
feature is intended as a temporary migration tool. We encourage the use
of CUE unification instead. After migration, the valueFiles field
should be refactored to the values field as one unified structure in
CUE. The valueFiles field makes this second order migration easier
becuase we can inspect and verify the complete rendered output, allowing
us to determine if a value is actually used in the final configuration
or is overridden.
[helm hierarchies]: https://medium.com/containers-101/using-helm-hierarchies-in-multi-source-argo-cd-applications-for-promoting-to-different-gitops-133c3bc93678
PROBLEM:
The Helm Values tutorial contains a fair bit of code/scripts, and we
need a way to test the steps we recommend to make sure nothing breaks
or slips out of date.
SOLUTION:
* Use `testscript` as a way to automate the execution of the steps in the doc and verify that none of the steps produce errors.
* Update the MDX file to directly reference the files embedded into the testscript.
OUTCOME:
* We have an automated way to perform the steps in the Helm Values document.
* We have unit tests that will fail should any of the commands being executed in the doc fail.
* The doc's MDX file directly references the files within the testscript, so we only need to modify the MDX file to update wording.
Without this patch selectors don't work as expected. This patch
fixes selectors such that each --selector flag value configures one
selector containing multiple positive or negative label matchers.
Result:
Render build plans for cluster dev or cluster test. Note the use of two
flags indicating logical OR.
holos render platform --selector cluster=test --selector cluster=dev
rendered external-secrets for cluster test in 299.897542ms
rendered external-secrets for cluster dev in 299.9225ms
rendered external-secrets-crds for cluster test in 667.6075ms
rendered external-secrets-crds for cluster dev in 708.126541ms
rendered platform in 708.795625ms
Render build plans for prod clusters that are not customer facing. Note
the use of one selector with comma separated labels.
holos render platform --selector "tier=prod,scope!=customer"
When new container image versions are built, automatically update the
holos-run/holos-action to use the new version.
Users of the action automatically update by default as a result.
When new container image versions are built, automatically update the
holos-run/holos-action to use the new version.
Users of the action automatically update by default as a result.
We need a released tag to reference in workflows that use the container
image to render the platform configuration.
This is the first image, subsequent git tags will also build and publish
container images.
Problem: We can't build old tags because the wrong Dockerfile is used
from the old tag.
Solution: Save the Dockerfile from main and use it to build the tag.
This create a dirty working directory but that's OK.
Push it to ghcr and quay.
* sign images with cosign and oidc id token
* add kustomize v5.5.0 to tools for distroless image
Usage:
docker run -v $(pwd):/app -w /app --rm -it ghcr.io/holos-run/holos:v0.101.8 holos render platform
The lint workflow was slow and we don't often change buf or angular
these days so they're not necessary.
The remaining valuable task is cspell, which we can speed up with a
dedicated step.
mpvl suggests @embed is a more ideal solution than our implementation of
core.Component.Instances for the use case of unifying YAML data updated
by Kargo Stage resources.
See the issue for a link to the discussion.
