Previously the Command core type was only useful for Validators and was
a bit hacky in that Holos appends the directory to the end of the
argument vector.
This patch changes the Command type to represent a generic command for
use as a Generator, Transformer, or Validator. The type is extended to
support the semantics of reading output from a file, directory, or
standard output pipe.
Go templates are used to fill in Holos managed data values, for example
the temporary directory associated with the pipeline task. The TaskData
structu represents these values passed into the template engine.
Previously using make bump to bump a version did not also update all of
the test cases and documentation to reflect the new version. This patch
updates the make bump tasks call HOLOS_UPDATE_SCRIPTS=1 scripts/test to
keep the test cases and documentation in sync with the new version.
Previously the tests fail because they were not updated to use the new
version string in holos, or the new topo sort behavior in cue 0.12.0.
This patch updates the test cases using:
HOLOS_UPDATE_SCRIPTS=1 scripts/test
Result: make test passes
PROBLEM:
The "Kustomize" tutorial has hardcoded code blocks and hasn't been
updated to use the automated testscript workflow.
SOLUTION:
Create a test for the Kustomize tutorial.
Create a testscript for the Kustomize test.
Update the Kustomize MDX file to load in data from the testscript directory.
OUTCOME:
The code content in the Kustomize tutorial now comes directly from the
testscript workflow.
PROBLEM:
The "Hello Holos" tutorial has hardcoded code blocks and hasn't been
updated to use the automated testscript workflow.
SOLUTION:
* Create a test for the Hello Holos tutorial.
* Create a testscript for the Hello Holos test.
* Update the Hello Holos MDX file to load in data from the testscript directory.
OUTCOME:
The code content in the Hello Holos tutorial now comes directly from the
testscript workflow.
* Convert all files with.period.separators to hyphen-separators.
* Rename and markdown_test.go to be specific to Helm Values.
* Move helm-values_test.go to be in the same directory as the Helm Values doc.
* Move Blackbox common configuration CUE file to `config/prometheus` so it can be imported as necessary.
* Use explicit import statements for Blackbox common config in `blackbox` and `prometheus` components.
Closes: #399
Without this patch migrating from [helm hierarchies] to Holos requires
the user to unify the value hierarchy. This is a problem because helm
hierarchies are difficult to unify because it's not clear if or why a
value is used in the final results. This makes it difficult to identify
how to resolve conflicts.
This patch adds `valueFiles` field to the Helm component kind. This
field is intended to provide a direct migration path from the
ApplicationSet.spec.template.spec.sources.helm.valueFiles field. With
this patch, users can directly migrate the values files to CUE using
`@embed`, then directly migrate the valueFiles field to reference the
values from within CUE.
Note we actively discourage the use of Helm value hierarchies. The
feature is intended as a temporary migration tool. We encourage the use
of CUE unification instead. After migration, the valueFiles field
should be refactored to the values field as one unified structure in
CUE. The valueFiles field makes this second order migration easier
becuase we can inspect and verify the complete rendered output, allowing
us to determine if a value is actually used in the final configuration
or is overridden.
[helm hierarchies]: https://medium.com/containers-101/using-helm-hierarchies-in-multi-source-argo-cd-applications-for-promoting-to-different-gitops-133c3bc93678
PROBLEM:
The Helm Values tutorial contains a fair bit of code/scripts, and we
need a way to test the steps we recommend to make sure nothing breaks
or slips out of date.
SOLUTION:
* Use `testscript` as a way to automate the execution of the steps in the doc and verify that none of the steps produce errors.
* Update the MDX file to directly reference the files embedded into the testscript.
OUTCOME:
* We have an automated way to perform the steps in the Helm Values document.
* We have unit tests that will fail should any of the commands being executed in the doc fail.
* The doc's MDX file directly references the files within the testscript, so we only need to modify the MDX file to update wording.
Previously holos render platform was not setting the --extract-yaml file
when calling holos render component, causing data file instances defined
in the Platform spec to be discarded.
This patch passes the value along using the flag.
Previously there isn't a good way to unify json and yaml files with the
cue configuration. This is a problem for use cases where data can be
generated idempotentialy prior to rendering the platform configuration.
The first use case is to explore unifying configuration with decrypted
sops values, which isn't typical since Holos is designed to handle
secrets with ExternalSecret resources, but does fit into the use case of
executing a command to produce data idempotently, then make the data
available to the platform configuration.
Other use cases this feature is intended to support are the prior
experiment where we fetch top level platform configuration from an rpc
service, and the future goal of integrating with data provided by
Terraform.
PROBLEM:
We've noticed that Holos almost immediately gets compared to Timoni, and
we frequently get asked for specifics in how they're similar/different.
SOLUTION:
* Add a `Comparison` page.
* Include a section that compares Holos to Timoni
OUTCOME:
Fewer questions about how Holos compares to Timoni because people are
able to find that answer themselves on our docs page.
It didn't work, failed with:
❯ holos show buildplans --selector app.holos.run/city=ams
could not run: Component.Name: 2 errors in empty disjunction: (and 2 more errors) at internal/builder/instance.go:66
Component.Name: 2 errors in empty disjunction:
Component.Name: conflicting values "no-name" and "podinfo-ams":
/Users/jeff/Holos/foo/holos-environments-tutorial/components/podinfo/podinfo.cue:6:12
/Users/jeff/Holos/foo/holos-environments-tutorial/schema.cue:6:13
/Users/jeff/Holos/foo/holos-environments-tutorial/schema.cue:35:2
/Users/jeff/Holos/foo/holos-environments-tutorial/tags.cue:13:19
Component.Name: conflicting values "podinfo" and "podinfo-ams":
/Users/jeff/Holos/foo/holos-environments-tutorial/components/podinfo/podinfo.cue:6:12
/Users/jeff/Holos/foo/holos-environments-tutorial/components/podinfo/podinfo.cue:7:8
/Users/jeff/Holos/foo/holos-environments-tutorial/schema.cue:6:13
/Users/jeff/Holos/foo/holos-environments-tutorial/schema.cue:35:2
This was likely because the podinfo component was used in different ways
in different topics. Don't use the shared component to fix the problem.
Previously holos unconditionally executed helm repo add which failed for
private repositories requiring basic authentication.
This patch addresses the problem by using the Helm SDK to pull and cache
charts without adding them as repositories. New fields for the
core.Helm type allow basic auth credentials to be read from environment
variables.
Multiple repositories are supported by using different env vars for
different repositories.
PROBLEM:
We've created a YouTube video walking people through Holos and the Helm
Values tutorial, but now we need to embed it on the site for visitors to
watch.
SOLUTION:
* Create a `YouTube` MDX plugin
* Use that plugin on Overview and Helm Values
* Tune the video size/attributes using CSS
OUTCOME:
The Helm Values YouTube video is embedded on our site for visitors to
watch.
Without this patch the validator fails if a component manages two of the
same kind of resource, which is common.
This patch updates the example to use the metadata namespace and name as
lookup keys. This works for most components, but may not for
ClusterResources. Use the kind top level field in that case and pass
the field name of the validator as a tag value to vary by component.
Similar to the Clusters topic, add a topic about configuring multiple
environments. This likely needs some work, the example is a bit
contrivied but at least shows how we can look up attributes, then use
those attributes to look up additional configuration from platform-wide
configuration data.
This commit removes the extra `./platform` argument from any of the
current tutorial/topic docs to reflect the change that was made to
`holos render platform` in version `0.100.0`.
Without this patch the BuildPlan resulting from a Platform that has
components with labels and annotations does not have the labels or
annotations of the source component.
Holos should copy the labels and annotations defined on each of the
Platform.spec.components to the resulting BuildPlan so end users can see
clearly where a BuildPlan originated from, and filter with selectors the
intermediate output BuildPlan the same way we filter with selectors the
original Platform spec components list.
Result:
```
holos init platform v1alpha5 --force
holos show buildplans | head
```
```yaml
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: podinfo
labels:
app.holos.run/cluster: local
app.holos.run/name: podinfo
annotations:
app.holos.run/description: podinfo for cluster local
```
Sometimes, but not always, the holos show buildplans command produces no
output.
```
❯ holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug
finalized config from flags
rendered platform in 13.458µs
```
It only happens when there's a selector. It doesn't happen without the
selector flag. It only happens with ==, not with =.
This test fails quickly.
```
while [[ $(holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug | wc -l) -eq 39 ]]; do true; done
```
This test runs until killed.
```
while [[ $(holos show buildplans --log-level=debug | wc -l) -eq 279 ]]; do true; done
```
Solution:
The problem is the use of the map. Iterating over the keys happens in a
random order. With the fix we check in an explicit order.
Show subcommand:
This is large change that accomplishes a number of goals. First, there
was no convenient way to show a build plan without using the debug logs
to indentify the tags to inject, then calling the cue command with the
right incantation to inspect the BuildPlan.
This patch addresses the problem by adding a `holos show buildplans`
command. The command loads the Platform spec from the platform
directory, then iterates over all Components to produce the BuildPlan.
This patch adds labels and annotations to the platform Components
collection in order to select and filter the output.
Result:
```
❯ holos show components --selector app.holos.run/cluster=local --format=yaml | head
kind: BuildPlan
apiversion: v1alpha5
metadata:
name: podinfo
spec:
artifacts:
- artifact: clusters/local/components/podinfo/podinfo.gen.yaml
generators:
- kind: Helm
output: helm.gen.yaml
```
---
Interface refactor:
This refactors the interface between the `holos` Go CLI layer and the
various core schema data structures. We now use a proper Go interface.
Concurrent execution over platform components has been improved to
accept a closure function so we can use the same interface method to
process the components. We use this to show each component and render
each component from different subcommands using the same interface
embedded in the builder.Platform struct.
The embedded interface allows us to easily swap in different versions,
e.g. v1beta1 and eventually v1. The number of interface methods are
quite small. 14 methods across 4 interfaces in holos/interface.go.
---
Remove old versions:
This patch removes support for versions prior to v1alpha5 in an effort
to clean up cruft.
Previously we didn't have good documentation on how to manage multiple
sets of clusters.
This patch adds a clusters topic in the structures category. Each one
of the environments, projects, owners, etc... structures follow the same
pattern as #Clusters and #ClusterSets, so it makes sense to put them
into a dedicated sidebar category for specific CUE structures.
When we moved from v1alpha4 to v1alpha5 we removed ArgoConfig from the
author schema. There was no longer a clear example of how to configure
an ArgoCD Application for every component in v1alpha5.
This patch adds a topic document with an example of how to add an
Application along side the resources by mixing an additional Artifact
into the BuildPlan.
Previously the Helm generator had no support for the --kube-version
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Previously the Helm generator had no support for the --api-versions
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Version doesn't make as much sense since we're doing a hello world
tutorial.
Also consolidate the values information into one step, and consolidate
the breaking it down section to make it shorter and clearer.
