Previously the compare yaml command was not registered, and invalid
subcommands returned exit code 0. Now invalid subcommands properly
return exit code 1 while help commands continue to return 0.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Fixed regression in holos render platform command where output manifests were
not being written to the deploy/ directory by default. Issues:
1. Ensured write-to flag is passed from render platform to render component
2. Fixed NewConfig to set the default WriteTo value properly
3. Fixed compile.go to ensure WriteTo is passed to components
4. Properly pass tempDir in renderAlpha5 method
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This command reads Component objects from a reader, exports a BuildPlan
from CUE, then marshals the result to the writer.
The purpose is to run concurrent instances of CUE to speed up build plan
generation prior to task execution.
- Add boolean flag --backwards-compatible (default false) to control backwards compatibility behavior
- Flag enables backwards compatibility mode where the second file may have fields missing from the first file
- Maintains existing behavior when flag is not set (strict comparison)
- Help text clearly explains the flag's purpose
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The BuildPlans function signature was updated to include an isBackwardsCompatible
boolean parameter, but the CLI command was not updated to provide this parameter.
Fixed by passing false for now to maintain existing behavior.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Move compare functionality from CLI package to dedicated internal/compare
package for better separation of concerns. CLI now acts as a thin wrapper
calling the compare package implementation.
- Create internal/compare package with Comparer type
- Move tests from CLI to compare package
- Update test fixtures path to testdata directory
- Replace CLI error with errors.NotImplemented()
- Clean up redundant test files from CLI package
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Introduce a new compare command with buildplans subcommand to verify
semantic equivalence between BuildPlan files. This will help users
validate their configurations remain equivalent when migrating from
v1alpha5 to v1alpha6.
- Add compare command structure with buildplans subcommand
- Implement table-driven test infrastructure reading from fixtures
- Create first test case with empty YAML files that expects failure
- Command returns "not implemented" error as a placeholder
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This patch changes the behavior of all commands to execute in the
current working directory of the platform root cue module. This gets
cue vet working well, otherwise it complains about fully qualified
paths. The cue command line expects the current working directory to be
a cue module.
The Env field of the Command schema wasn't implemented, so we remove it
from the schema. It's unclear it's necessary. Setting the environment
in the parent context should suffice for current use cases.
Testing is problematic because the current working directory is not the
platform root. This patch refactors the codebase to consistently store
and use the platform root directory to construct absolute paths for
reading and writing files during a BuldPlan execution.
There isn't good test coverage of the v1alpha6 component package
responsible for producing and executing a BuildPlan from a Component.
This patch adds test coverage using a shared testutils package using the
generate package to generate a v1alpha6 platform module, then copying
fixtures from the embedded testutils.Fixtures fs.FS.
Name it consistently, Context can be a bit ambiguous. Also define the
tags used to pass data from the platform layer to the component layer as
go constants in the core api for clear documentation.
Previously we were having to define selectors on every command that
processes platform components. This patch fixes the problems by moving
component selectors to the platform.Config struct.
Result:
Commands that process platform components need to explicity add the
platform.Config.FlagSet to the command, but otherwise the configuration
is consolidated and reused as we want. The holos show platform command
always uses the default configuration so we don't bother adding the
flags for this case.
This patch refactors the holos render platform command to use the new
platform package instead of the previous builder package. Similar to
the holos show buildplans command, the PerComponentFunc is used, the
main difference is it simply constructs an argument vector to execute
holos render component injecting the proper flags and tags for cue to
produce the build plan.
Previously the test coverage for the holos show buildplans command was
covered by testscript. This is a problem because it's difficult to do a
structural comparison in testscript, we're limited to comparing stdout
with a file.
This patch refactors the test script to a go testing test using testify
to compare structures instead of comparing text output as was done
previously.
In the process it became clear the holos show buildplans command is not
injecting tag values properly. The component name, labels, and
annotations were missing. This patch fixes that problem by passing the
tags from the core Platform.Components element to the
component.Component.BuildPlan method.
Note, I'm not happy with how ad-hoc the tags have become to pass around.
Ideally we'll refactor them into a proper protocol, passing the entire
core.Component structure to the method that produces a build plan. The
idea of passing this structure over standard intput to a build plan task
worker keeps coming up over an over again.
Previously the show buildplans, render platform, and render component
commands used duplicate code paths to obtain build plans and render
components. This patch consolidates the behaviors into the component
and platform packages. The platform Build method is used to call a
different PerComponentFunc depending on the use case of showing build
plans or rendering each component in the platform.
The behavior of loading a build plan from cue is consolidated into the
Component.BuildPlan method.
Result:
1. Platform.Build is reused for both show buildplans and render platform
2. Component.BuildPlan is reused for both show buildplans and render
component.
There are only two version specific cue builders, a Platform and a
component BuildPlan. The platform is anemic while the component holds
most of the behavior of the Holos rendering pipeline. Without this
patch the two types of builders were mixed together in the same package.
This patch refactors the version specific builders into the platform and
component packages respectively. The version specific BuildPlan builder
is where most of the behavior lies and what we expect to evolve most
over time. With the typemeta.yaml approach, we can easily switch code
paths, isolated to the process boundary since holos render platform
invokes sub processes for each holos render component command.
This patch helps focus on good test coverage by establishing a clear
area to add version specific test cases, both for the platform layer and
the individual component layer.
This patch uses the same platform embedded into the holos init platform
v1alpha6 subcommand for the test cases to exercise backwards
compatibility.
This is prep work to refactor the builder logic into the component
package for better organization and test coverage.
The minimal test case is an empty build plan with only an apiVersion and
a kind field. This patch adds such a test case with the typemeta.yaml
file and enough cue code to get the cue.Value and load it into a Go
struct for holos to process.
Result:
The test case passes and provides a starting point to add additional
coverage.
go test -coverprofile=coverage.out ./internal/cli/render/component/...
ok github.com/holos-run/holos/internal/cli/render/component 0.612s coverage: 46.7% of statements
There hasn't been good unit level test coverage of the component
rendering behavior. This patch refactors the behavior of the holos
render component command into a component package with a primary
Component struct.
The result is the package is better organized for unit level testing
across api versions, notably v1alpha6 with the new arbitrary Command
generators, transformers, and validators.
Previously, holos loaded the full CUE Instance to discriminate against
the APIVersion and Kind. This is a problem because we need to know the
api version to know what tags to inject. For example, v1alpha6 needs to
have the build context injected but v1alpha5 does not.
This patch fixes the problem by changing holos render component to look
for a `typemeta.yaml` file in the component directory. The command line
tool discriminates on the kind and apiVersion specified in this file
prior to building the CUE instance.
Result:
We're able to inject version specific cue tags into the CUE instance.
Note previously the build context was passed to CUE by writing a file to
the filesystem and using the embed functionality. We cannot proceed
with this embed approach because the filesystem writes are not safe for
concurrent use. Therefore, we inject the build context as a tag which
is safe for concurrent use.
Without this patch selectors don't work as expected. This patch
fixes selectors such that each --selector flag value configures one
selector containing multiple positive or negative label matchers.
Result:
Render build plans for cluster dev or cluster test. Note the use of two
flags indicating logical OR.
holos render platform --selector cluster=test --selector cluster=dev
rendered external-secrets for cluster test in 299.897542ms
rendered external-secrets for cluster dev in 299.9225ms
rendered external-secrets-crds for cluster test in 667.6075ms
rendered external-secrets-crds for cluster dev in 708.126541ms
rendered platform in 708.795625ms
Render build plans for prod clusters that are not customer facing. Note
the use of one selector with comma separated labels.
holos render platform --selector "tier=prod,scope!=customer"
I'd like to add the kargo-demo repository to Unity to test evalv3, but
can't get a handle on the main function to wire up to testscript.
This patch fixes the problem by moving the MakeMain function to a public
package so the kargo-demo go module can import and call it using the go
mod tools technique.
Previously holos render platform was not setting the --extract-yaml file
when calling holos render component, causing data file instances defined
in the Platform spec to be discarded.
This patch passes the value along using the flag.
Extract YAML is more clear and aligns with the schema docs for the
Component Instance field which has an extractYAML kind. This also
leaves the door open for additional kinds of data extractors which are
almost certainly going to be needed.
Previously there isn't a good way to unify json and yaml files with the
cue configuration. This is a problem for use cases where data can be
generated idempotentialy prior to rendering the platform configuration.
The first use case is to explore unifying configuration with decrypted
sops values, which isn't typical since Holos is designed to handle
secrets with ExternalSecret resources, but does fit into the use case of
executing a command to produce data idempotently, then make the data
available to the platform configuration.
Other use cases this feature is intended to support are the prior
experiment where we fetch top level platform configuration from an rpc
service, and the future goal of integrating with data provided by
Terraform.
Previously, build tags were not propagated from `holos render platform
-t validate` through to the underlying `holos render component` command.
This is a problem because validators need to be selectively enabled as a
work around until we have an audit mode field.
This patch fixes the problem by propagating command line tags from the
render platform command to the underlying commands. This patch also
propagates tags for the show command.
Previously Holos only supported tags in the form of key=value. CUE
supports boolean style tags in the form of `key [ "=" value ]` which we
want to use to conditionally use to register components with the
platform.
This patch modifies the flag parsing to support -t foo like cue does,
for use with the @if(foo) build tag.
Writes files based on parent pid and process pid to avoid collisions.
Analyze with:
export HOLOS_TRACE=trace.%d.%d.out
go tool trace trace.999.1000.out
export HOLOS_CPU_PROFILE=cpu.%d.%d.prof
go tool pprof cpu.999.1000.prof
export HOLOS_MEM_PROFILE=mem.%d.%d.prof
go tool pprof mem.999.1000.prof
Without this patch `holos cue vet` always returns exit code 0, even when
there are errors.
This patch fixes the problem by catching the error and returning it to
our own top level error handler. Note the final error, "could not run:
terminating because of errors" which wraps the generic error reported by
cue in the presence of multiple errors.
Result:
```
❯ holos cue vet ./policy --path 'strings.ToLower(kind)' /tmp/podinfo.gen.yaml
deployment.kind: conflicting values "Forbidden" and "Deployment":
./policy/validations.cue:18:8
../../../../../tmp/podinfo.gen.yaml:25:7
deployment.spec.template.spec.containers.0.resources.limits: conflicting values null and {[string]:"k8s.io/apimachinery/pkg/api/resource".#Quantity} (mismatched types null and struct):
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:355:9
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:376:12
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2840:11
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2968:14
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:15
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:18
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:5027:9
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:6407:16
./policy/validations.cue:17:13
../../../../../tmp/podinfo.gen.yaml:104:19
could not run: terminating because of errors
```
Without this patch the BuildPlan resulting from a Platform that has
components with labels and annotations does not have the labels or
annotations of the source component.
Holos should copy the labels and annotations defined on each of the
Platform.spec.components to the resulting BuildPlan so end users can see
clearly where a BuildPlan originated from, and filter with selectors the
intermediate output BuildPlan the same way we filter with selectors the
original Platform spec components list.
Result:
```
holos init platform v1alpha5 --force
holos show buildplans | head
```
```yaml
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: podinfo
labels:
app.holos.run/cluster: local
app.holos.run/name: podinfo
annotations:
app.holos.run/description: podinfo for cluster local
```
Without this patch the holos show buildplans command returns results in
an inconsistent order. This is a problem because the output should be
idempotent.
This patch fixes the problem by adding an EncodeSeq(idx int, v any) method to
the encoder interface. idx represents the index position of the
Platform.spec.components list after selector filtering has been applied.
This patch modifies the json and yaml encoders to buffer out of order
results from the concurrent go routines.
Result:
Concurrent execution is preserved. The buffer is kept to a reasonable
size, entries are deleted once they're encoded in the correct order.
Most importantly the output is consistent and idempotent so we can write
effective integration tests.
Sometimes, but not always, the holos show buildplans command produces no
output.
```
❯ holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug
finalized config from flags
rendered platform in 13.458µs
```
It only happens when there's a selector. It doesn't happen without the
selector flag. It only happens with ==, not with =.
This test fails quickly.
```
while [[ $(holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug | wc -l) -eq 39 ]]; do true; done
```
This test runs until killed.
```
while [[ $(holos show buildplans --log-level=debug | wc -l) -eq 279 ]]; do true; done
```
Solution:
The problem is the use of the map. Iterating over the keys happens in a
random order. With the fix we check in an explicit order.
Show subcommand:
This is large change that accomplishes a number of goals. First, there
was no convenient way to show a build plan without using the debug logs
to indentify the tags to inject, then calling the cue command with the
right incantation to inspect the BuildPlan.
This patch addresses the problem by adding a `holos show buildplans`
command. The command loads the Platform spec from the platform
directory, then iterates over all Components to produce the BuildPlan.
This patch adds labels and annotations to the platform Components
collection in order to select and filter the output.
Result:
```
❯ holos show components --selector app.holos.run/cluster=local --format=yaml | head
kind: BuildPlan
apiversion: v1alpha5
metadata:
name: podinfo
spec:
artifacts:
- artifact: clusters/local/components/podinfo/podinfo.gen.yaml
generators:
- kind: Helm
output: helm.gen.yaml
```
---
Interface refactor:
This refactors the interface between the `holos` Go CLI layer and the
various core schema data structures. We now use a proper Go interface.
Concurrent execution over platform components has been improved to
accept a closure function so we can use the same interface method to
process the components. We use this to show each component and render
each component from different subcommands using the same interface
embedded in the builder.Platform struct.
The embedded interface allows us to easily swap in different versions,
e.g. v1beta1 and eventually v1. The number of interface methods are
quite small. 14 methods across 4 interfaces in holos/interface.go.
---
Remove old versions:
This patch removes support for versions prior to v1alpha5 in an effort
to clean up cruft.
Previously the holos render platform and component subcommands had flags
for oidc authentication and client access to the gRPC service. These
flags aren't currently used, they're remnants from the json powered form
prototype.
This patch gates the flags behind a feature flag which is disabled by
default.
Result:
holos render platform --help
render an entire platform
Usage:
holos render platform DIRECTORY [flags]
Examples:
holos render platform ./platform
Flags:
--concurrency int number of components to render concurrently (default 8)
-v, --version version for platform
Global Flags:
--log-drop strings log attributes to drop (example "user-agent,version")
--log-format string log format (text|json|console) (default "console")
--log-level string log level (debug|info|warn|error) (default "info")
---
HOLOS_FEATURE_CLIENT=1 holos render platform --help
render an entire platform
Usage:
holos render platform DIRECTORY [flags]
Examples:
holos render platform ./platform
Flags:
--concurrency int number of components to render concurrently (default 8)
--oidc-client-id string oidc client id. (default "270319630705329162@holos_platform")
--oidc-extra-scopes strings optional oidc scopes
--oidc-force-refresh force refresh
--oidc-issuer string oidc token issuer url. (default "https://login.holos.run")
--oidc-scopes strings required oidc scopes (default openid,email,profile,groups,offline_access)
--server string server to connect to (default "https://app.holos.run:443")
-v, --version version for platform
Global Flags:
--log-drop strings log attributes to drop (example "user-agent,version")
--log-format string log format (text|json|console) (default "console")
--log-level string log level (debug|info|warn|error) (default "info")
Previously the Helm generator had no support for the --api-versions
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
