Cloudflare Pages looks for .nvmrc at the project root before running
the build command, not in subdirectories.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add .nvmrc file to specify Node 22 for Cloudflare Pages builds.
Update engines in package.json to require Node >= 20.0 to support
dependabot updates that require at least Node 20.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add kubectl to tools.go to pin the version and allow installation via
`go install k8s.io/kubectl`. This upgrades Kubernetes-related
dependencies from v0.33.3 to v0.34.3.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Install github.com/rogpeppe/go-internal/cmd/testscript when running
make tools for script-based testing support.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Jeff McCune <jeff@openinfrastructure.co>
It's really good at filtering resources like Namespace, Secret, and
CustomResourceDefinition. It's excellent at breaking up files into an
easy to navigate, clear structure that integrates well with tools. fzf
for example.
Use v.Validate(cue.Concrete(true)) before decoding to get user-friendly
field paths like "holos.metadata.name" instead of internal type names
like "#Metadata.name". This makes error messages clearer and matches
the output style of cue export.
With this patch the error is as I want it to be:
❯ holos show platform
error at internal/platform/platform.go:153: holos.metadata.name: incomplete value string
holos.metadata.name: incomplete value string:
/Users/jeff/Holos/holos/tmp/examples/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha6/types_go_gen.cue:419:8
- Updates the tested Helm version to v3.17.3 in the setup documentation.
- Adds a "Known Issues" section to the setup documentation.
- Notes that Helm v3.18.0 produces incorrect output and links to the relevant Holos and Nixpkgs issues.
Fixes#433
Update testscript files to use `holos compare yaml` instead of `cmp`
for YAML file comparisons. This enables structural comparison rather
than textual comparison, ensuring semantic equivalence instead of
character-by-character matching.
Changes:
- Replace cmp stdout with cp stdout + holos compare yaml pattern
- Update all test files that compare YAML content
- Maintain test functionality while using semantic comparison
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Previously the compare yaml command was not registered, and invalid
subcommands returned exit code 0. Now invalid subcommands properly
return exit code 1 while help commands continue to return 0.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
When executing in the test harness, we cannot assume the os.Executable
is the holos executable, otherwise we recursively execute the
go-build1600354897/b1280/cli.test compile command forever.
Makes it easier to test and know for certain where the value is being
set from. Previously it wasn't clear how the write to directory was
being modified or read in the global config object.
Fixed regression in holos render platform command where output manifests were
not being written to the deploy/ directory by default. Issues:
1. Ensured write-to flag is passed from render platform to render component
2. Fixed NewConfig to set the default WriteTo value properly
3. Fixed compile.go to ensure WriteTo is passed to components
4. Properly pass tempDir in renderAlpha5 method
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This package clarifies and simplifies the input protocol of the holos
compile command. A BuildPlanRequest represents the complete context
necessary to compile a BuildPlan.
This patch also upgrades cue to v0.13.0-alpha4
This command reads Component objects from a reader, exports a BuildPlan
from CUE, then marshals the result to the writer.
The purpose is to run concurrent instances of CUE to speed up build plan
generation prior to task execution.
- Add detailed explanation of what isBackwardsCompatible controls
- Include example showing how newer versions can have additional fields
- Clarify that fields in before must always be present in after
- Make the documentation more user-friendly to prevent confusion
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Replace 'a' with 'before' and 'b' with 'after' in BuildPlans doc comment for clarity
- Fix backwards compatibility test case by swapping before/after files:
- before.yaml now correctly has fewer fields (older version)
- after.yaml now correctly has more fields (newer version with enhancements)
- Update comparison logic to filter fields from after that don't exist in before
- This correctly implements spec 4: after may have fields missing from before when backwards compatible
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add boolean flag --backwards-compatible (default false) to control backwards compatibility behavior
- Flag enables backwards compatibility mode where the second file may have fields missing from the first file
- Maintains existing behavior when flag is not set (strict comparison)
- Help text clearly explains the flag's purpose
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add test case BuildPlan_4 for BuildPlan spec 4: b may have fields missing from a if isBackwardsCompatible is true
- Implement backwards compatibility logic in comparison functions
- Add filterToCommonFields() to filter fields based on what exists in both structures
- Update compareDocumentLists() and compareStructures() to accept isBackwardsCompatible parameter
- When backwards compatible mode is enabled, only compare fields that exist in both structures
- Update test framework to support isBackwardsCompatible field in test cases
- Test verifies that missing labels, annotations, and nested fields don't cause failure in backwards compatible mode
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
The BuildPlans function signature was updated to include an isBackwardsCompatible
boolean parameter, but the CLI command was not updated to provide this parameter.
Fixed by passing false for now to maintain existing behavior.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Reference BuildPlan spec items in test names (BuildPlan_3.1, BuildPlanFile_1, etc.)
- Add missing test cases for BuildPlan spec requirements:
- BuildPlanFile_2: each object matches exactly one unique object
- BuildPlan_5: fields in 'a' must exist in 'b'
- BuildPlan_6.1-6.3: null/empty/missing field equivalence
- BuildPlanValid_1-2: duplicate objects and same metadata.name
- Implement null/empty/missing field equivalence (spec 6)
- Add normalizeStructure() to handle spec 6 requirements
- Ensure null, [], and missing fields are treated as equivalent
- Update test messages to clarify spec coverage intent
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add comprehensive test cases to cover all specification requirements:
- File spec requirement 1: equal number of BuildPlans
- BuildPlan spec 1&2: all fields must be equivalent
- BuildPlan spec 3: nested objects must match recursively
- BuildPlan spec 4: exact field value matching with exceptions
- BuildPlan spec 4.1: unordered spec.artifacts list
- BuildPlan spec 4.2: key order irrelevance
- Toleration 1: multiple identical BuildPlans treated as unique
- Toleration 2: multiple BuildPlans with same metadata.name
Update existing test names and messages to clearly reference the specific specification requirements they validate.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Remove all label-specific references and simplify the diff output to show only what go-cmp reports. Extract field differences for cleaner test error messages while keeping the implementation generic for all fields.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This patch adds the output of the new stdio concurrenet compiler being
developed for v1alpha6 as after, which is semantically equivalent to the
v0.104.1 v1alpha5 holos show buildplans output.
Note the field orders and relative object order in the stream is quite
different.
- Remove all label-specific functions and logic
- Treat labels as regular nested fields with no special handling
- Extract field differences from go-cmp output for test compatibility
- Remove before/after YAML output, show only the extracted differences
- All tests pass with this simplified, generic implementation
- Remove all hardcoded references to seq label
- Extract all labels for comparison instead of handling seq specially
- Compare structures without labels first, then check label differences
- Replace seq-specific functions with generic label handling functions
- This makes the implementation generic for any arbitrary labels
- Implement bipartite matching algorithm to match documents
- First pass finds exact matches including all labels
- Second pass handles unmatched documents and reports seq differences
- This allows labels2 (reordered documents) to pass without errors
- This allows labels3 (changed seq values) to report seq differences
The key insight is that labels3 has a document with seq:2 that has no
exact match in the after file (which only has seq:1 documents), so we
need to report the seq difference for that case.
- Sort documents by composite key before comparison to handle out-of-order objects
- Exclude transient labels (like 'seq') from composite key generation
- Add deepCopy method to avoid modifying original structures
- Add removeTransientLabels to exclude labels that shouldn't affect semantic equivalence
- Use copies of documents in compareStructures to preserve original data
- Support comparison of identical objects that appear in different order in streams
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Remove hardcoded test case checks for holos.run/stack.name and buildplan4
- Rename extractTestErrors to extractDifferences for clarity
- Make implementation completely generic to extract any field additions
- Maintain backward compatibility by keeping the same output format
- Remove all test-specific knowledge from the compare package
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Refactor test infrastructure to support expectedErrors as a list instead of single expectedError
- Implement generic field difference extraction for any object additions
- Update extractTestErrors method to handle arbitrary field additions
- Migrate all test cases from expectedError to expectedErrors format
- Maintain backward compatibility with existing test expectations
- Add support for detecting and formatting any field additions in diff output
- Clean up go-cmp type annotations in error messages for better readability
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enhance error messages to show the full diff output from go-cmp along
with specific field errors. This provides better visibility into all
differences between BuildPlans when they don't match.
- Show both specific field errors and full diff
- Maintain backward compatibility with existing tests
- Provide comprehensive diff output for debugging
- Improve user experience when comparing BuildPlans
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Add support for showing meaningful diff messages when BuildPlans don't
match. Extract specific field differences from go-cmp diff output to
provide targeted error messages for common mismatches.
- Use go-cmp with custom transformers for order-independent comparison
- Parse diff output to extract specific field differences
- Show targeted error messages for label mismatches
- Pass buildplan3 test for different stack names
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Extend comparison logic to handle full BuildPlans with out-of-order
artifacts. Implement recursive deep comparison that sorts arrays at
all levels for true order-independent equality checking.
- Add deepEqual method for recursive order-independent comparison
- Improve sortSlice to handle nested maps correctly
- Support comparing complex BuildPlans with reordered artifacts
- Pass buildplan2 test for full BuildPlan comparison
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Extend the composite key function to handle BuildPlans with identical
names but different labels. This ensures proper matching when BuildPlans
have the same metadata.name but differ in other metadata fields.
- Enhance getCompositeKey to include sorted label key-value pairs
- Support BuildPlans with duplicate names distinguished by labels
- Ensure order-independent comparison works with complex metadata
- Pass streams3 test case for duplicate names with different labels
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement the name field for all test cases to provide more descriptive
test names in the test output. This improves test readability and
maintenance.
- Add name field to testCase struct
- Update test logic to use name field when provided
- Add descriptive names to all test cases
- Maintain nested test structure with directory/name hierarchy
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enhance test cases with descriptive messages that explain what each
test is verifying. These messages are displayed when tests fail,
improving debugging experience.
- Add msg field to testCase struct
- Update all testcase.json files with descriptive messages
- Use messages in test assertions for better failure output
- Ensure all tests continue to pass with new structure
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Switch to using google/go-cmp for more robust comparison logic that
handles unordered document lists. This allows BuildPlans to be
considered equivalent even when documents appear in different orders.
- Replace custom equalMaps with cmp.Equal
- Add order-independent comparison for document lists
- Use cmpopts.SortSlices with composite key for sorting
- Support streams2 test case with reordered documents
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Extend BuildPlan comparison to handle YAML streams containing multiple
documents separated by ---. The implementation now properly parses and
compares files with multiple BuildPlan objects.
- Add parseYAMLStream function to handle multi-document YAML files
- Add compareDocumentLists to compare lists of parsed documents
- Refactor single document parsing to use streaming approach
- Maintain order-sensitive comparison for document sequences
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement basic BuildPlan comparison functionality that can handle the
minimal test case. The implementation reads and parses YAML files,
then compares them for semantic equivalence.
- Add file reading and YAML parsing logic
- Implement basic map comparison for BuildPlans
- Handle empty files case with NotImplemented error
- Use errors.Format for consistent error handling
- Support minimal BuildPlans with just version and kind fields
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Simplify test case directory name from empty-files to empty.
- Rename testdata/empty-files/ to testdata/empty/
- Test continues to pass with cleaner naming
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Rename test configuration files from want.json to testcase.json for
better clarity of purpose. Updated test code to match the new naming.
- Rename want.json to testcase.json in testdata
- Update variable names from wantData to testcaseData
- Update error messages to reference testcase.json
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Move compare functionality from CLI package to dedicated internal/compare
package for better separation of concerns. CLI now acts as a thin wrapper
calling the compare package implementation.
- Create internal/compare package with Comparer type
- Move tests from CLI to compare package
- Update test fixtures path to testdata directory
- Replace CLI error with errors.NotImplemented()
- Clean up redundant test files from CLI package
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Introduce a new compare command with buildplans subcommand to verify
semantic equivalence between BuildPlan files. This will help users
validate their configurations remain equivalent when migrating from
v1alpha5 to v1alpha6.
- Add compare command structure with buildplans subcommand
- Implement table-driven test infrastructure reading from fixtures
- Create first test case with empty YAML files that expects failure
- Command returns "not implemented" error as a placeholder
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This patch changes the behavior of all commands to execute in the
current working directory of the platform root cue module. This gets
cue vet working well, otherwise it complains about fully qualified
paths. The cue command line expects the current working directory to be
a cue module.
The Env field of the Command schema wasn't implemented, so we remove it
from the schema. It's unclear it's necessary. Setting the environment
in the parent context should suffice for current use cases.
Dry up the tests using a helper function to iterate over a simple
struct, they're all executing a component and validating the rendered
manifests against an expected data file.
Previously the command and kustomize transformers were implemented as a
stand alone function. The current method is to implement tasks as
methods on the generator, transformer, or validator structs. This patch
refactors the stand alone functions to methods for consistency.
It also adds test coverage for the command and kustomize transformers.
Testing is problematic because the current working directory is not the
platform root. This patch refactors the codebase to consistently store
and use the platform root directory to construct absolute paths for
reading and writing files during a BuldPlan execution.
There isn't good test coverage of the v1alpha6 component package
responsible for producing and executing a BuildPlan from a Component.
This patch adds test coverage using a shared testutils package using the
generate package to generate a v1alpha6 platform module, then copying
fixtures from the embedded testutils.Fixtures fs.FS.
Good to finally kill this one off, the behavior has moved to the
platform and component packages using an embedded interface to abstract
different APIVersions.
Name it consistently, Context can be a bit ambiguous. Also define the
tags used to pass data from the platform layer to the component layer as
go constants in the core api for clear documentation.
Previously we were having to define selectors on every command that
processes platform components. This patch fixes the problems by moving
component selectors to the platform.Config struct.
Result:
Commands that process platform components need to explicity add the
platform.Config.FlagSet to the command, but otherwise the configuration
is consolidated and reused as we want. The holos show platform command
always uses the default configuration so we don't bother adding the
flags for this case.
This patch refactors the holos render platform command to use the new
platform package instead of the previous builder package. Similar to
the holos show buildplans command, the PerComponentFunc is used, the
main difference is it simply constructs an argument vector to execute
holos render component injecting the proper flags and tags for cue to
produce the build plan.
Previously the test coverage for the holos show buildplans command was
covered by testscript. This is a problem because it's difficult to do a
structural comparison in testscript, we're limited to comparing stdout
with a file.
This patch refactors the test script to a go testing test using testify
to compare structures instead of comparing text output as was done
previously.
In the process it became clear the holos show buildplans command is not
injecting tag values properly. The component name, labels, and
annotations were missing. This patch fixes that problem by passing the
tags from the core Platform.Components element to the
component.Component.BuildPlan method.
Note, I'm not happy with how ad-hoc the tags have become to pass around.
Ideally we'll refactor them into a proper protocol, passing the entire
core.Component structure to the method that produces a build plan. The
idea of passing this structure over standard intput to a build plan task
worker keeps coming up over an over again.
Previously the show buildplans, render platform, and render component
commands used duplicate code paths to obtain build plans and render
components. This patch consolidates the behaviors into the component
and platform packages. The platform Build method is used to call a
different PerComponentFunc depending on the use case of showing build
plans or rendering each component in the platform.
The behavior of loading a build plan from cue is consolidated into the
Component.BuildPlan method.
Result:
1. Platform.Build is reused for both show buildplans and render platform
2. Component.BuildPlan is reused for both show buildplans and render
component.
There are only two version specific cue builders, a Platform and a
component BuildPlan. The platform is anemic while the component holds
most of the behavior of the Holos rendering pipeline. Without this
patch the two types of builders were mixed together in the same package.
This patch refactors the version specific builders into the platform and
component packages respectively. The version specific BuildPlan builder
is where most of the behavior lies and what we expect to evolve most
over time. With the typemeta.yaml approach, we can easily switch code
paths, isolated to the process boundary since holos render platform
invokes sub processes for each holos render component command.
This patch helps focus on good test coverage by establishing a clear
area to add version specific test cases, both for the platform layer and
the individual component layer.
This patch uses the same platform embedded into the holos init platform
v1alpha6 subcommand for the test cases to exercise backwards
compatibility.
This is prep work to refactor the builder logic into the component
package for better organization and test coverage.
The minimal test case is an empty build plan with only an apiVersion and
a kind field. This patch adds such a test case with the typemeta.yaml
file and enough cue code to get the cue.Value and load it into a Go
struct for holos to process.
Result:
The test case passes and provides a starting point to add additional
coverage.
go test -coverprofile=coverage.out ./internal/cli/render/component/...
ok github.com/holos-run/holos/internal/cli/render/component 0.612s coverage: 46.7% of statements
There hasn't been good unit level test coverage of the component
rendering behavior. This patch refactors the behavior of the holos
render component command into a component package with a primary
Component struct.
The result is the package is better organized for unit level testing
across api versions, notably v1alpha6 with the new arbitrary Command
generators, transformers, and validators.
Previously, holos loaded the full CUE Instance to discriminate against
the APIVersion and Kind. This is a problem because we need to know the
api version to know what tags to inject. For example, v1alpha6 needs to
have the build context injected but v1alpha5 does not.
This patch fixes the problem by changing holos render component to look
for a `typemeta.yaml` file in the component directory. The command line
tool discriminates on the kind and apiVersion specified in this file
prior to building the CUE instance.
Result:
We're able to inject version specific cue tags into the CUE instance.
Note previously the build context was passed to CUE by writing a file to
the filesystem and using the embed functionality. We cannot proceed
with this embed approach because the filesystem writes are not safe for
concurrent use. Therefore, we inject the build context as a tag which
is safe for concurrent use.
Previously, the build plan CUE code has no way to refer to the temporary
directory managed and owned by the `holod render component` command.
This patch adds a placeholder context field for holos to fill in just
before exporting the final build plan from CUE.
This allows the user to refer to the tempDir field without the value
being concrete, useful for constructing an argument vector for the new
Command task.
Without this patch the holos render platform command fails:
holos init platform v1alpha6
holos render platform
could not run: unsupported version: v1alpha6 at internal/builder/platform.go:102
This patch copies the v1alpha5 builder implementation to the v1alpha6 go
package and adds a switch case handler for Platform v1alpha6 apiVersions
Result:
holos render platform
rendered platform in 15.459µs
Without this patch the default apiVersion for the Platform resource is
v1alpha5 even though we're using the v1alpha6 schemas. This patch
updates the default value and re-generates the docs and cue schema using
`make generate`
Before:
holos show platform
apiVersion: v1alpha5
kind: Platform
metadata:
name: default
spec:
components: []
Result:
holos init platform v1alpha6 --force
holos show platform
apiVersion: v1alpha6
kind: Platform
metadata:
name: default
spec:
components: []
Rendering the platform fails as expected:
holos render platform
could not run: unsupported version: v1alpha6 at internal/builder/platform.go:102
Without this patch the holos render platform command fails with the
following error. This patch adds the additional type definitions in CUE
to make the metadata.name field concrete.
This patch copies the pkg directory from the `cue.mod` directory using
`rsync -avxHP v1alpha5/ v1alpha6/` for both the core and author schemas.
Result:
holos init platform v1alpha6
holos render platform
rendered platform in 33.917µs
Without this patch the v1alpha6 of the author schema imports v1alpha5 of
the core schema. This patch imports the v1alpha6 core schema.
This is a necessary step to get holos render platform working after
holos init platform v1alpha6.
Result: We still have other things to copy over to establish v1alpha6.
holos render platform
could not run: holos.metadata.name: cannot convert non-concrete value string at builder/v1alpha5/builder.go:34
holos.metadata.name: cannot convert non-concrete value string:
/Users/jeff/Holos/command/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha6/types_go_gen.cue:305:2
Without this patch init fails with:
holos init platform v1alpha6
could not run: cannot generate: have: [v1alpha6] want: [v1alpha4 v1alpha5] at internal/generate/platform.go:68
This patch copies the v1alpha5 initial platform to establish v1alpha6.
Result: holos init platform v1alpha6 works.
Without this patch the docs website has a conflict over the v1alpha5
path. This patch fixes the problem by setting the current version to
v1alpha6.
Result: npm run build in doc/website now builds successfully. All links
automatically route to v1alpha6 url paths. The previous v1alpha5 paths
are accessible.
[INFO] [en] Creating an optimized production build...
[WARNING] Tags [launch] used in 2024-10-28-announcing-holos.mdx are not defined in tags.yml
[WARNING] Tags [cue] used in 2024-10-28-why-cue.md are not defined in tags.yml
[WARNING] Tags [feature] used in 2024-11-25-validators.mdx are not defined in tags.yml
[WARNING] No docs found in "api/author": can't auto-generate a sidebar.
[WARNING] No docs found in "api/core": can't auto-generate a sidebar.
Generate docs from the v1alpha6 schemas to make sure the website and
links are correct.
This results in the following warnings about conflicting routes when
running `npm run build` in doc/website
[INFO] [en] Creating an optimized production build...
[WARNING] Tags [cue] used in 2024-10-28-why-cue.md are not defined in tags.yml
[WARNING] Tags [feature] used in 2024-11-25-validators.mdx are not defined in tags.yml
[WARNING] Tags [launch] used in 2024-10-28-announcing-holos.mdx are not defined in tags.yml
[WARNING] No docs found in "api/author": can't auto-generate a sidebar.
[WARNING] No docs found in "api/core": can't auto-generate a sidebar.
[WARNING] Duplicate routes found!
- Attempting to create page at /docs/v1alpha5/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/api/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/api/author/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/api/core/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/common/example-component-integrate/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/common/example-component/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/glossary/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/support/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/architecture/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/comparison/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/gitops/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/gitops/argocd-application/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/gitops/flux-kustomization/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/kargo/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/local-cluster/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/oci-helm-charts/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/private-helm/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/structures/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/structures/clusters/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/topics/structures/environments/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/tutorial/cue/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/tutorial/hello-holos/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/tutorial/helm-values/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/tutorial/kustomize/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/tutorial/overview/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/tutorial/setup/, but a page already exists at this route.
- Attempting to create page at /docs/v1alpha5/tutorial/validators/, but a page already exists at this route.
This could lead to non-deterministic routing behavior.
In preparation for v1alpha6 where we're designing a generic Command task
for use in generators, transformers, and validators, we need to create
permalinks for the existing v1alpha5 schemas.
This patch copies the current documentation to the v1alpha5 versioned
docs links and establishes v1alpha6 as the current version.
See: https://github.com/holos-run/holos/pull/413 for context.
Without this patch there are unexpected lint errors in version 1.60
where 1.61.0 passes locally on my machine.
This patch updates to:
golangci-lint has version 1.64.5 built with go1.24.0 from 0a603e49 on 2025-02-13T21:19:55Z
Previously using make bump to bump a version did not also update all of
the test cases and documentation to reflect the new version. This patch
updates the make bump tasks call HOLOS_UPDATE_SCRIPTS=1 scripts/test to
keep the test cases and documentation in sync with the new version.
Previously the tests fail because they were not updated to use the new
version string in holos, or the new topo sort behavior in cue 0.12.0.
This patch updates the test cases using:
HOLOS_UPDATE_SCRIPTS=1 scripts/test
Result: make test passes
PROBLEM:
The "Kustomize" tutorial has hardcoded code blocks and hasn't been
updated to use the automated testscript workflow.
SOLUTION:
Create a test for the Kustomize tutorial.
Create a testscript for the Kustomize test.
Update the Kustomize MDX file to load in data from the testscript directory.
OUTCOME:
The code content in the Kustomize tutorial now comes directly from the
testscript workflow.
PROBLEM:
The "Hello Holos" tutorial has hardcoded code blocks and hasn't been
updated to use the automated testscript workflow.
SOLUTION:
* Create a test for the Hello Holos tutorial.
* Create a testscript for the Hello Holos test.
* Update the Hello Holos MDX file to load in data from the testscript directory.
OUTCOME:
The code content in the Hello Holos tutorial now comes directly from the
testscript workflow.
* Convert all files with.period.separators to hyphen-separators.
* Rename and markdown_test.go to be specific to Helm Values.
* Move helm-values_test.go to be in the same directory as the Helm Values doc.
* Move Blackbox common configuration CUE file to `config/prometheus` so it can be imported as necessary.
* Use explicit import statements for Blackbox common config in `blackbox` and `prometheus` components.
Closes: #399
Without this patch migrating from [helm hierarchies] to Holos requires
the user to unify the value hierarchy. This is a problem because helm
hierarchies are difficult to unify because it's not clear if or why a
value is used in the final results. This makes it difficult to identify
how to resolve conflicts.
This patch adds `valueFiles` field to the Helm component kind. This
field is intended to provide a direct migration path from the
ApplicationSet.spec.template.spec.sources.helm.valueFiles field. With
this patch, users can directly migrate the values files to CUE using
`@embed`, then directly migrate the valueFiles field to reference the
values from within CUE.
Note we actively discourage the use of Helm value hierarchies. The
feature is intended as a temporary migration tool. We encourage the use
of CUE unification instead. After migration, the valueFiles field
should be refactored to the values field as one unified structure in
CUE. The valueFiles field makes this second order migration easier
becuase we can inspect and verify the complete rendered output, allowing
us to determine if a value is actually used in the final configuration
or is overridden.
[helm hierarchies]: https://medium.com/containers-101/using-helm-hierarchies-in-multi-source-argo-cd-applications-for-promoting-to-different-gitops-133c3bc93678
PROBLEM:
The Helm Values tutorial contains a fair bit of code/scripts, and we
need a way to test the steps we recommend to make sure nothing breaks
or slips out of date.
SOLUTION:
* Use `testscript` as a way to automate the execution of the steps in the doc and verify that none of the steps produce errors.
* Update the MDX file to directly reference the files embedded into the testscript.
OUTCOME:
* We have an automated way to perform the steps in the Helm Values document.
* We have unit tests that will fail should any of the commands being executed in the doc fail.
* The doc's MDX file directly references the files within the testscript, so we only need to modify the MDX file to update wording.
Without this patch selectors don't work as expected. This patch
fixes selectors such that each --selector flag value configures one
selector containing multiple positive or negative label matchers.
Result:
Render build plans for cluster dev or cluster test. Note the use of two
flags indicating logical OR.
holos render platform --selector cluster=test --selector cluster=dev
rendered external-secrets for cluster test in 299.897542ms
rendered external-secrets for cluster dev in 299.9225ms
rendered external-secrets-crds for cluster test in 667.6075ms
rendered external-secrets-crds for cluster dev in 708.126541ms
rendered platform in 708.795625ms
Render build plans for prod clusters that are not customer facing. Note
the use of one selector with comma separated labels.
holos render platform --selector "tier=prod,scope!=customer"
When new container image versions are built, automatically update the
holos-run/holos-action to use the new version.
Users of the action automatically update by default as a result.
When new container image versions are built, automatically update the
holos-run/holos-action to use the new version.
Users of the action automatically update by default as a result.
We need a released tag to reference in workflows that use the container
image to render the platform configuration.
This is the first image, subsequent git tags will also build and publish
container images.
Problem: We can't build old tags because the wrong Dockerfile is used
from the old tag.
Solution: Save the Dockerfile from main and use it to build the tag.
This create a dirty working directory but that's OK.
Push it to ghcr and quay.
* sign images with cosign and oidc id token
* add kustomize v5.5.0 to tools for distroless image
Usage:
docker run -v $(pwd):/app -w /app --rm -it ghcr.io/holos-run/holos:v0.101.8 holos render platform
The lint workflow was slow and we don't often change buf or angular
these days so they're not necessary.
The remaining valuable task is cspell, which we can speed up with a
dedicated step.
mpvl suggests @embed is a more ideal solution than our implementation of
core.Component.Instances for the use case of unifying YAML data updated
by Kargo Stage resources.
See the issue for a link to the discussion.
I'd like to add the kargo-demo repository to Unity to test evalv3, but
can't get a handle on the main function to wire up to testscript.
This patch fixes the problem by moving the MakeMain function to a public
package so the kargo-demo go module can import and call it using the go
mod tools technique.
Previously holos render platform was not setting the --extract-yaml file
when calling holos render component, causing data file instances defined
in the Platform spec to be discarded.
This patch passes the value along using the flag.
Extract YAML is more clear and aligns with the schema docs for the
Component Instance field which has an extractYAML kind. This also
leaves the door open for additional kinds of data extractors which are
almost certainly going to be needed.
Previously there isn't a good way to unify json and yaml files with the
cue configuration. This is a problem for use cases where data can be
generated idempotentialy prior to rendering the platform configuration.
The first use case is to explore unifying configuration with decrypted
sops values, which isn't typical since Holos is designed to handle
secrets with ExternalSecret resources, but does fit into the use case of
executing a command to produce data idempotently, then make the data
available to the platform configuration.
Other use cases this feature is intended to support are the prior
experiment where we fetch top level platform configuration from an rpc
service, and the future goal of integrating with data provided by
Terraform.
PROBLEM:
We've noticed that Holos almost immediately gets compared to Timoni, and
we frequently get asked for specifics in how they're similar/different.
SOLUTION:
* Add a `Comparison` page.
* Include a section that compares Holos to Timoni
OUTCOME:
Fewer questions about how Holos compares to Timoni because people are
able to find that answer themselves on our docs page.
It didn't work, failed with:
❯ holos show buildplans --selector app.holos.run/city=ams
could not run: Component.Name: 2 errors in empty disjunction: (and 2 more errors) at internal/builder/instance.go:66
Component.Name: 2 errors in empty disjunction:
Component.Name: conflicting values "no-name" and "podinfo-ams":
/Users/jeff/Holos/foo/holos-environments-tutorial/components/podinfo/podinfo.cue:6:12
/Users/jeff/Holos/foo/holos-environments-tutorial/schema.cue:6:13
/Users/jeff/Holos/foo/holos-environments-tutorial/schema.cue:35:2
/Users/jeff/Holos/foo/holos-environments-tutorial/tags.cue:13:19
Component.Name: conflicting values "podinfo" and "podinfo-ams":
/Users/jeff/Holos/foo/holos-environments-tutorial/components/podinfo/podinfo.cue:6:12
/Users/jeff/Holos/foo/holos-environments-tutorial/components/podinfo/podinfo.cue:7:8
/Users/jeff/Holos/foo/holos-environments-tutorial/schema.cue:6:13
/Users/jeff/Holos/foo/holos-environments-tutorial/schema.cue:35:2
This was likely because the podinfo component was used in different ways
in different topics. Don't use the shared component to fix the problem.
Previously holos unconditionally executed helm repo add which failed for
private repositories requiring basic authentication.
This patch addresses the problem by using the Helm SDK to pull and cache
charts without adding them as repositories. New fields for the
core.Helm type allow basic auth credentials to be read from environment
variables.
Multiple repositories are supported by using different env vars for
different repositories.
PROBLEM:
We've created a YouTube video walking people through Holos and the Helm
Values tutorial, but now we need to embed it on the site for visitors to
watch.
SOLUTION:
* Create a `YouTube` MDX plugin
* Use that plugin on Overview and Helm Values
* Tune the video size/attributes using CSS
OUTCOME:
The Helm Values YouTube video is embedded on our site for visitors to
watch.
Without this patch we do not support installing Kargo from an OCI helm
chart. We want to support:
```
Component: #Helm & {
Name: "kargo"
Namespace: Kargo.Namespace
Chart: {
name: "oci://ghcr.io/akuity/kargo-charts/kargo"
version: "1.0.3"
release: Name
}
EnableHooks: true
Values: Kargo.Values
}
```
This patch fixes the problem by using the base name for filesystem cache
operations.
This shows up in the Unity tests I'm working on with mvdan and goes to a
blank page without the redirect in place.
--- FAIL: TestGuides_v1alpha5 (0.00s)
--- FAIL: TestGuides_v1alpha5/helm (0.60s)
testscript.go:584: # Helm Guide https://holos.run/docs/guides/helm/
Previously, build tags were not propagated from `holos render platform
-t validate` through to the underlying `holos render component` command.
This is a problem because validators need to be selectively enabled as a
work around until we have an audit mode field.
This patch fixes the problem by propagating command line tags from the
render platform command to the underlying commands. This patch also
propagates tags for the show command.
Previously Holos only supported tags in the form of key=value. CUE
supports boolean style tags in the form of `key [ "=" value ]` which we
want to use to conditionally use to register components with the
platform.
This patch modifies the flag parsing to support -t foo like cue does,
for use with the @if(foo) build tag.
Previously the BuildPlan pipeline didn't execute generators and
transformers concurrently. All steps were sequentially executed. Holos
was primarily concurrent by executing multiple BuildPlans at once.
This patch changes the Build implementation for each BuildPlan to
execute a GoRoutine pipeline. One producer fans out to a group of
routines each executing the pipeline for one artifact in the build plan.
The pipeline has 3 stages:
1: Fan-out to build each Generator concurrently.
2: Fan-in to build each Transformer sequentially.
3: Fan-out again to run each validator concurrently.
When the artifact pipelines return, the producer closes the tasks
channel causing the worker tasks to return.
Note the overall runtime for 8 BuildPlans is roughly equivalent to
previously at 160ms with --concurrency=8 on my M3 Max. I expect this to
perform better than previously when multiple artifacts are rendered for
each BuildPlan.
Writes files based on parent pid and process pid to avoid collisions.
Analyze with:
export HOLOS_TRACE=trace.%d.%d.out
go tool trace trace.999.1000.out
export HOLOS_CPU_PROFILE=cpu.%d.%d.prof
go tool pprof cpu.999.1000.prof
export HOLOS_MEM_PROFILE=mem.%d.%d.prof
go tool pprof mem.999.1000.prof
Without this patch the validator fails if a component manages two of the
same kind of resource, which is common.
This patch updates the example to use the metadata namespace and name as
lookup keys. This works for most components, but may not for
ClusterResources. Use the kind top level field in that case and pass
the field name of the validator as a tag value to vary by component.
Without this patch `holos cue vet` always returns exit code 0, even when
there are errors.
This patch fixes the problem by catching the error and returning it to
our own top level error handler. Note the final error, "could not run:
terminating because of errors" which wraps the generic error reported by
cue in the presence of multiple errors.
Result:
```
❯ holos cue vet ./policy --path 'strings.ToLower(kind)' /tmp/podinfo.gen.yaml
deployment.kind: conflicting values "Forbidden" and "Deployment":
./policy/validations.cue:18:8
../../../../../tmp/podinfo.gen.yaml:25:7
deployment.spec.template.spec.containers.0.resources.limits: conflicting values null and {[string]:"k8s.io/apimachinery/pkg/api/resource".#Quantity} (mismatched types null and struct):
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:355:9
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:376:12
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2840:11
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2968:14
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:15
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:18
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:5027:9
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:6407:16
./policy/validations.cue:17:13
../../../../../tmp/podinfo.gen.yaml:104:19
could not run: terminating because of errors
```
Similar to the Clusters topic, add a topic about configuring multiple
environments. This likely needs some work, the example is a bit
contrivied but at least shows how we can look up attributes, then use
those attributes to look up additional configuration from platform-wide
configuration data.
This commit removes the extra `./platform` argument from any of the
current tutorial/topic docs to reflect the change that was made to
`holos render platform` in version `0.100.0`.
If someone accidentally provides the same index multiple times, or
indexes less than the next expected, the program would silently discard
the data. This would be difficult to troubleshoot since an
OrderedEncoder is usually used with concurrent go routines, which would
likely mislead the investigator.
Better to just fail hard with an error indicating the caller in these
situations.
Assert against the complete build plan so we know if we change the
output format in the future.
It's easy to update if so:
HOLOS_UPDATE_SCRIPTS=1 go test github.com/holos-run/holos/cmd/holos
Without this patch trying to use a Kustomize patch with the optional
name field omitted results in the following error:
could not run: holos.spec.artifacts.0.transformers.0.kustomize.kustomization.patches.0.target.name: cannot convert non-concrete value string at builder/v1alpha5/builder.go:218
holos.spec.artifacts.0.transformers.0.kustomize.kustomization.patches.0.target.name: cannot convert non-concrete value string:
$WORK/cue.mod/gen/sigs.k8s.io/kustomize/api/types/var_go_gen.cue:33:2
This patch fixes the problem by providing a default value for the name
field matching the Go zero value for a string.
Without this patch the BuildPlan resulting from a Platform that has
components with labels and annotations does not have the labels or
annotations of the source component.
Holos should copy the labels and annotations defined on each of the
Platform.spec.components to the resulting BuildPlan so end users can see
clearly where a BuildPlan originated from, and filter with selectors the
intermediate output BuildPlan the same way we filter with selectors the
original Platform spec components list.
Result:
```
holos init platform v1alpha5 --force
holos show buildplans | head
```
```yaml
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: podinfo
labels:
app.holos.run/cluster: local
app.holos.run/name: podinfo
annotations:
app.holos.run/description: podinfo for cluster local
```
Without this patch the holos show buildplans command returns results in
an inconsistent order. This is a problem because the output should be
idempotent.
This patch fixes the problem by adding an EncodeSeq(idx int, v any) method to
the encoder interface. idx represents the index position of the
Platform.spec.components list after selector filtering has been applied.
This patch modifies the json and yaml encoders to buffer out of order
results from the concurrent go routines.
Result:
Concurrent execution is preserved. The buffer is kept to a reasonable
size, entries are deleted once they're encoded in the correct order.
Most importantly the output is consistent and idempotent so we can write
effective integration tests.
Sometimes, but not always, the holos show buildplans command produces no
output.
```
❯ holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug
finalized config from flags
rendered platform in 13.458µs
```
It only happens when there's a selector. It doesn't happen without the
selector flag. It only happens with ==, not with =.
This test fails quickly.
```
while [[ $(holos show buildplans --selector app.holos.run/cluster==w3 --log-level=debug | wc -l) -eq 39 ]]; do true; done
```
This test runs until killed.
```
while [[ $(holos show buildplans --log-level=debug | wc -l) -eq 279 ]]; do true; done
```
Solution:
The problem is the use of the map. Iterating over the keys happens in a
random order. With the fix we check in an explicit order.
Without this patch the `holos show buildplans` BuildPlan output has
incorrect yaml with the v3 encoder. For example apiversion: v1alpha5
instead of apiVersion.
Show subcommand:
This is large change that accomplishes a number of goals. First, there
was no convenient way to show a build plan without using the debug logs
to indentify the tags to inject, then calling the cue command with the
right incantation to inspect the BuildPlan.
This patch addresses the problem by adding a `holos show buildplans`
command. The command loads the Platform spec from the platform
directory, then iterates over all Components to produce the BuildPlan.
This patch adds labels and annotations to the platform Components
collection in order to select and filter the output.
Result:
```
❯ holos show components --selector app.holos.run/cluster=local --format=yaml | head
kind: BuildPlan
apiversion: v1alpha5
metadata:
name: podinfo
spec:
artifacts:
- artifact: clusters/local/components/podinfo/podinfo.gen.yaml
generators:
- kind: Helm
output: helm.gen.yaml
```
---
Interface refactor:
This refactors the interface between the `holos` Go CLI layer and the
various core schema data structures. We now use a proper Go interface.
Concurrent execution over platform components has been improved to
accept a closure function so we can use the same interface method to
process the components. We use this to show each component and render
each component from different subcommands using the same interface
embedded in the builder.Platform struct.
The embedded interface allows us to easily swap in different versions,
e.g. v1beta1 and eventually v1. The number of interface methods are
quite small. 14 methods across 4 interfaces in holos/interface.go.
---
Remove old versions:
This patch removes support for versions prior to v1alpha5 in an effort
to clean up cruft.
Previously the holos render platform and component subcommands had flags
for oidc authentication and client access to the gRPC service. These
flags aren't currently used, they're remnants from the json powered form
prototype.
This patch gates the flags behind a feature flag which is disabled by
default.
Result:
holos render platform --help
render an entire platform
Usage:
holos render platform DIRECTORY [flags]
Examples:
holos render platform ./platform
Flags:
--concurrency int number of components to render concurrently (default 8)
-v, --version version for platform
Global Flags:
--log-drop strings log attributes to drop (example "user-agent,version")
--log-format string log format (text|json|console) (default "console")
--log-level string log level (debug|info|warn|error) (default "info")
---
HOLOS_FEATURE_CLIENT=1 holos render platform --help
render an entire platform
Usage:
holos render platform DIRECTORY [flags]
Examples:
holos render platform ./platform
Flags:
--concurrency int number of components to render concurrently (default 8)
--oidc-client-id string oidc client id. (default "270319630705329162@holos_platform")
--oidc-extra-scopes strings optional oidc scopes
--oidc-force-refresh force refresh
--oidc-issuer string oidc token issuer url. (default "https://login.holos.run")
--oidc-scopes strings required oidc scopes (default openid,email,profile,groups,offline_access)
--server string server to connect to (default "https://app.holos.run:443")
-v, --version version for platform
Global Flags:
--log-drop strings log attributes to drop (example "user-agent,version")
--log-format string log format (text|json|console) (default "console")
--log-level string log level (debug|info|warn|error) (default "info")
Previously we didn't have good documentation on how to manage multiple
sets of clusters.
This patch adds a clusters topic in the structures category. Each one
of the environments, projects, owners, etc... structures follow the same
pattern as #Clusters and #ClusterSets, so it makes sense to put them
into a dedicated sidebar category for specific CUE structures.
When we moved from v1alpha4 to v1alpha5 we removed ArgoConfig from the
author schema. There was no longer a clear example of how to configure
an ArgoCD Application for every component in v1alpha5.
This patch adds a topic document with an example of how to add an
Application along side the resources by mixing an additional Artifact
into the BuildPlan.
Previously the Helm generator had no support for the --kube-version
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Previously the Helm generator had no support for the --api-versions
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Version doesn't make as much sense since we're doing a hello world
tutorial.
Also consolidate the values information into one step, and consolidate
the breaking it down section to make it shorter and clearer.
npm i @docusaurus/core@latest @docusaurus/plugin-client-redirects@latest \
@docusaurus/preset-classic@latest @docusaurus/theme-mermaid@latest \
@docusaurus/module-type-aliases@latest @docusaurus/tsconfig@latest \
@docusaurus/types@latest
This time in the correct directory.
A tree view of the `holos-tutorial/` directory should give readers a
quick, high-level understanding of the folder structure of a typical
Holos platform project.
Previously the current version would always be unreleased at /docs/next
and we'd have to copy the doc/md/ folder into the
doc/website/versioned_docs/version-v1alpha5/ every time we made a
change.
We're going to be working on v1alpha5 for awhile so it makes sense to
just have the current version published at /docs/v1alpha5/ and we can
start /docs/v1alpha6/ whenever we're ready.
This also has the nice effect of giving us permalinks if we change the
structure again. /docs/v1alpha5/ will remain over time.
Add a helm values tutorial which is a cut down version of the v1alpha4
helm guide. The httpbin kustomize will immediately follow building on
the prometheus and blackbox charts.
Previously the holos command line expected a Platform and BuildPlan
resource at the top level of the exported data from CUE. This forced us
to use hidden fields for everything else.
This patch modifies the BuildData struct to first look for a holos top
level field and use it if present. This opens up other top level fields
for use by end users.
Our intent is to reserve any top level field prefixed with holos.
Note this follows how Timoni works as well.
This patch strips down the v1alpha4 core and author schemas to only with
is absolutely necessary for all holos users. Aspects of platform
configuration applicable to some, even most, but not all users will be
moved into documentation topics organized as a recipe book.
The functionality removed from the v1alpha4 author schemas in v1alpha5
will move into self contained examples documented as topics on the docs
site.
The overall purpose is to have a focused, composeable, maintainable
author schema to help people get started and ideally we can support for
years with making breaking changes.
With this patch the v1alpha5 helm guide test passes. We're not going to
have this guide anymore but it demonstrates we're back to where we were
with v1alpha4.
Without this patch each version of the core and author schemas are
duplicated into each docs version. This is unnecessary and difficult to
maintain now that we have docusaurus versioned docs enabled.
This patch updates the schema generation script to check if the docs
version has been released, and if so write into a markdown file in the
versioned docs folder. If not, the version is written into the next
version folder.
This patch also updates some, but not all, document links to the md or
mdx relative file paths. This is necessary to generate the correct
versioned links.
A nice outcome of this change is that technical docs no longer need to
link to version specific pages. For example, `[Core Schema]:
./api/core.md` will always refer to the correct auto generated docs
associated with the docs version.
The docs for v1alpha4 have the right information, but in the wrong
places. The most important bits are tucked away in the Core API docs.
One of our first users entirely missed the `holos generate platform`
command mentioned in the Helm guide.
We'll fix this by organizing the docs into two distinct categories.
First, a tutorial written as a series progressively building up the
minimum knowledge to use holos effectively and gain the benefits. Think
of it as a tour of the essential bits.
The second category are focused topics which stand alone. They're the
things most people using holos will need to know eventually, but aren't
essential for everyone to know. For example, Clusters and Fleets will
move from the Author API to stand alone examples of how to implement
these features if necessary.
Then there's a Glossary which serves as the place to describe our
concepts and domain specific language.
Finally there's the API documentation which should be cut down to the
specific version. The next release version will be v1alpha5.
Attribution: We're copying the Tokio docs structure, it's concise and a
similar size and complexity to our own project.
The Go docs are also an inspiration, but the project is much larger so
not directly comparable. The organization of https://go.dev/doc/ feels
complete at first glance, despite the size and age of the project. The
site also makes clear who each section is for without needing to come
right out and say it. Getting started, Using and understanding Go,
Writing modules, using databases, etc...
We switched from using a kustomize remote base to a local file so the
tests don't need to make a network round trip to github. It's also
better practice to use local files for this sort of thing.
In doing so I botched the location of the file, putting it in the
platform registration section. This patch claifies how `resources.yaml`
is linked to `httpbin.cue` through the `KustomizeConfig: Files:
"resources.yaml": _` field.
Previously there was no test coverage of the
https://holos.run/docs/guides/helm/ guide. This patch uses Roger's
testscript package, which the CUE folks also use to add comprehensive
test coverage of each step in the guide. Ideally we would execute these
commands directly from the guide itself, but for now we'll duplicate the
commands into the test script. This could be enhanced by generating the
test script from the document itself in some way.
When updating the script, use the `holos txtar` command to embed entire
helm charts into the test script. It's not super fast, but it's better
than network access and it's not terribly slow either. A few seconds to
unpack.
---
txtar: quote files for testscript unquote
For the helm guide test script we want to include the entire helm chart
which may have files that need to be quoted. This patch changes the
default behavior of the holos txtar command to quote files if necessary
and list them in an unquote script command in the comment of the
archive.
The purpose is for testscript authors to copy and paste the entire thing
into a test script and include the unquote command at the top.
---
This change also updates CUE to v0.10.1
No longer necessary now that we're on v1alpha4. Test coverage for
v1alpha4 and the user facing guides will be added back soon for use both
in the holos repo and in Unity.
Previously it wasn't clear for users if platform wide structs should be
definitions or hidden fields in CUE. They should be hidden fields when
they contain data and definitions when they define a schema.
This patch updates the generate platform v1alpha4 subcommand to use the
correct field names consistently for clarity.
Generate the social card manually from https://www.opengraph.xyz/
Override the page title tag, otherwise it shows up as "Announcing Holos
| Holos" in social links, which is weird.
The api references are in reverse order and don't have good descriptions
in the index listings. This patch adds front matter to each generated
document to order them correctly and add a nice description.
Without this patch it's difficult to mix in a plain file as a config
map. This is necessary for the use case of using a Job to generate a
secret in-cluster. We want a plain shell script to be carried through
and transformed into the job.
We already have the KustomizeConfig fields to support this, they just
weren't wired up to the #Kustomization component kind.
I didn't check if it's wired up to Helm and Kustomize for expedience.
They may be missing there as well.
PROBLEM:
Version v1alpha4 of the Author API has been updated with backwards
incompatible changes, and the deploy-a-service guide uses code from
version v1alpha3.
SOLUTION:
Update any code, links, and data that is out of date, and then run
through the guide to make sure it works locally.
OUTCOME:
The instructions in the deploy-a-service guide will work successfully
with version v1alpha4 of the Author API.
Cue uses --inject, -t as the flags to set variables for fields tagged
using @tag(var,type=string).
We used --tag, which is different and requires a mental mapping. Let's
use the same flag and also pass it multiple times like they require so
we can copy and paste the command line output from the debug logs into a
cue export command to see what's going on.
This patch deprecates the --cluster-name flag, use --inject
holos_cluster=mycluster instead.
This patch also removes the environment field from the Component core
API, leaving this to the user namespace to define via tags. We don't
want to be too opinionated on how users manage their platform, baking
environment into the schema is a slippery slope toward those kinds of
opinions.
Closes: #276
Now that we have CommonLabels as part of the ComponentConfig for all
components, it makes sense to also mix in CommonLabels for a Project.
Common labes are key aspect of the Technical Overview document.
For the Author API, it would be nice to define a schema for the fields
common to all component kinds. Users could then configure all kinds by
unifying the schema into their own platform tree.
This makes a clear use case to extract the common fields back into an
embedded struct like we did in v1alpha3. I removed the embedded struct
in v1alpha4 because it wasn't clear why it should be separate, but now
the use case is clear, to configure all component kinds.
Without this patch holos render platform may hang until the overall
context timeout is reached. This is a problem because the user has no
idea why it's hung.
This patch adds a warning at the 5 second and another at the 10 second
mark indicating the lock may be deadlocked. The user can then remove
the directory.
The Kustomize build plan kind needs to support both copying files from
the component directory and pulling resources from https URL's. Without
this patch this support is missing from the Author API
With this patch the Kustomize build plan kind has a KustomizeConfig
field with two structs, Files and Resources. The kustomization
resources list is built up from both of these.
Two transformers are used so we don't affect the GitOps transfomer which
really only needs CommonLabels.
I decided to keep this field exclusive to the Kustomize kind, but it
could replace the Kustomization field of the other kinds as well.
Without this patch the user facing API doesn't have a way to kustomize
the output of all the build plan kinds. This patch ensures the
Kustomization field is present on all of Helm, Kustomize, and
Kubernetes.
This field is inteded for patches and transforms. The second
kustomization in the transformer sequence is intended for common labels
and annotations, managed by a corresponding field instead of a full on
Kustomization resource.
Fix:
could not run: could not marshal json projects/platform/components/cert-manager: cue: marshal error: spec.artifacts.0.generators.0.helm.enableHooks: cannot convert incomplete value "bool" to JSON at internal/builder/builder.go:63
spec.artifacts.0.generators.0.helm.enableHooks: cannot convert incomplete value "bool" to JSON:
/Users/jeff/Holos/bank-of-holos/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha4/types_go_gen.cue:235:16
could not run: could not render component: exit status 1 at builder/v1alpha4/builder.go:94
Without this patch kustomize errors aren't surfaced when executing holos
render platform.
This patch gives a fighting chance to the user to figure out what's
going on. The stderr is copied, logged, and surfaced up to the parent
holos render platform command.
Previously the #Helm and #Kustomize build plan helpers were not defined
in the v1alpha4 Author API. We need this definition to update the
Quickstart guide for v1alpha4 from v1alpha3.
This patch defines the #Helm and #Kustomize helpers in the Author API
similar to how #Kubernetes is defined.
Previously #Kubernetes was defined in the platform code. This is a
problem because every platform engineer would need to copy and paste
this code.
This patch moves the #Kubernetes helper into the cue.mod directory so it
can be imported and used ergonomically.
This patch gets the Author API rendering the namespaces component in the
Bank of Holos guide. It's not the final form of the API yet, we still
need to decide how best to expose the Kubernetes, Helm, and Kustomize
definitions.
I'm thinking we abstract away the transformers and generators within the
Author API Kubernetes definition.
Without this patch the --write-to flag can't be controlled from the
PlatformSpec in the CoreAPI. We need to surface this for the ArgoConfig
struct in the AuthorAPI.
That is to say, in v1alpha3 the --write-to flag was previously assumed
to be deploy/ in ArgoConfig using the DeployFiles functionality. We no
longer have DeployFiles in Core API v1alpha4, all artifacts are instead
written relative to the --write-to flag. Still, we need to expose this
flag in the PlatformSpec so users can use something other than the
deploy directory.
Previously the file generator was unimplemented. This patch implements
it as a simple file read into the ArtifactMap for use by the Kustomize
or Join transformers.
With this patch all v1alpha4 Core API features are implemented.
Resources, Helm, and File generators. Kustomize and Join transformers.
Blank lines show up in the output which is confusing. This patch fixes
the only source location identified with the following command.
export HOLOS_LOG_LEVEL=debug
export HOLOS_LOG_FORMAT=json
holos render platform ./platform 2>&1 | jq -r 'select (.msg == "")'
Previously helm charts were cached only by name, which is a problem
because the wrong version would be used when previously cached.
This patch caches charts by name and version to ensure changes in the
version results in pulling the new cached version. It is the user's
responsibility to remove old versions.
This patch also ensures only a single go routine can run cacheChart() at
a time across processes. This is necessary when rendering a platform
because multiple processes will run the Helm generator concurrently, for
example when the same chart is used for multiple environments or
customers.
The mkdir system call serves as the locking mechanism, which is robust
and atomic on all commonly used file systems.
Previously the helm generator was not implemented and returned an error.
This patch is a first pass copying the helm method from
internal/render/helm.go
Basic testing performed with a podinfo chart. It works as the previous
versions in v1alpha3 and before works. This patch does not address the
cached version issue in #273
holos.FilePath is intended for paths relative to the platform root
directory. We use the Artifact to store lots of stuff not related to
the platform root directory, for example kustomization.yaml in a temp
dir. Most entries are not relative to the platform root directory given
the implicit cfg.WriteTo prefix.
Previously:
could not run: could not build dev-join: could not get foo.yaml: not set at builder/v1alpha4/builder.go:180
This is confusing because set has nothing to do with the missing input
from the cue code the user writes.
Result:
could not run: could not build test-join: missing foo.yaml at builder/v1alpha4/builder.go:180
This is better because it at doesn't distract the user from the fact
they're missing a foo.yaml generator output to align with the
transformer input.
The code was inlined in a number of places, it makes sense to move it to
the interface. It'll also make it easier to test, we can provide a null
writer concrete value.
Previously the Artifact collection was processed sequentially. This
patch provides a modest performance improvement, about 16% faster for
our simple 2 artifact use case, by processing each artifact
concurrently.
Platform rendering provides poor user feedback:
```
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
```
We want to see the metadata.name field of each BuildPlan. This patch
injects the build plan name from the platform spec to make the name
available through the end to end platform rendering process.
Result:
```
❯ holos render platform ./platforms/minimal
rendered stage-namespaces for cluster local in 146.078375ms
rendered prod-namespaces for cluster local in 146.544583ms
rendered test-namespaces for cluster local in 147.0535ms
rendered dev-namespaces for cluster local in 147.499166ms
rendered platform in 147.553875ms
```
With this patch the first use case of CUE Resources + Kustomize is fully
working, artifacts are written into the deploy directory.
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
The output indicates we need to plumb the BuildPlan metadata.name from
the PlatfromSpec through to the render component command. This is
necessary so we can report the correct name instead of just the base
path.
Without this patch holos writes a single yaml document that is a list.
It needs to write a file that contains multiple documents, each document
a map[string]any representing the kubernetes resource.
This patch fixes the problem. With this patch kustomize fully executes.
The manifest field isn't clear.
Much more clear to have generators produce one Output. Transformers
take multiple Inputs and produce one Output.
The final Transformer, or a single Generator, must produce the final
Artifact.
The Inputs and Output naming to produce an Artifact makes clear the
rendering pipeline we're implementing.
This also makes clear that multiple generators must have at least one
transformer to produce the final output artifact. We model a simple
Join transformer for this case, which is what `holos` was implicitly
doing previously.
Component makes much more sense, that's the domain terminology we use.
BuildContext was meant to be re-used elsewhere, but we never did so the
name serves no purpose.
The repeated enabled booleans and file fields are awkward. It's clear
it's three separate things smashed into one.
kustomize isn't really a generator. It's useless because there is no
way to reference a plain file in a component directory.
This patch replaces the kustomize generator with a file generator which
simply reads one single file. Multiple of these generators may be used
to read one or more files.
Then, kustomize may transform these generated files, which are generated
by simply reading from the filesystem.
This API is much improved over the previous.
```
kind: BuildPlan
apiVersion: v1alpha4
metadata:
name: prod-namespaces
spec:
component: projects/platform/components/namespaces
steps:
- artifact: clusters/no-cluster/components/prod-namespaces/prod-namespaces.gen.yaml
generators:
- kind: Resources
manifest: resources.gen.yaml
resources:
Namespace:
prod-jeff:
metadata:
name: prod-jeff
labels:
kubernetes.io/metadata.name: prod-jeff
kind: Namespace
apiVersion: v1
prod-gary:
metadata:
name: prod-gary
labels:
kubernetes.io/metadata.name: prod-gary
kind: Namespace
apiVersion: v1
prod-nate:
metadata:
name: prod-nate
labels:
kubernetes.io/metadata.name: prod-nate
kind: Namespace
apiVersion: v1
transformers:
- kind: Kustomize
kustomize:
kustomization:
commonLabels:
holos.run/component.name: prod-namespaces
resources:
- resources.gen.yaml
- application.gen.yaml
- artifact: clusters/no-cluster/gitops/prod-namespaces.gen.yaml
generators:
- kind: Resources
manifest: application.gen.yaml
resources:
Application:
argocd:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: prod-namespaces
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
project: default
source:
path: examples/v1alpha4/deploy/clusters/no-cluster/components/prod-namespaces
repoURL: https://github.com/holos-run/bank-of-holos
targetRevision: main
transformers:
- kind: Kustomize
kustomize:
kustomization:
commonLabels:
holos.run/component.name: prod-namespaces
resources:
- resources.gen.yaml
- application.gen.yaml
```
A build step either produces kubernetes objects or a gitops manifest.
Both are effectively the same, they're just kubernetes resources.
For the use case of applying common labels to both, we'll have the
Author API pass the same Kustomization to two separate build steps. One
step to produce the resources, a second to produce the argocd
application or flux kustomization.
Each step produces a manifest and a gitops file, so we need a unique
name for each step. The most common case will be a single build step
matching the name of the build plan itself.
The kustomize transformer needs a filename to store the output from
generators so it has an input for the transformer. This patch adds
fields for each kind of generator so the kustomize.#Kustomization can be
configured with the files `holos` will write generated output to.
This patch implements the v1alpha4 component rendering builder for a
component BuildPlan. We don't yet have the CUE definitions, so this
hasn't been end to end tested yet, the next step is defining the
generators and transforms in the core API BuildPlan.
This patch plumbs the switch statement to branch on a v1alpha4
BuildPlan. Tags need to be passed from the render platform subcommand
to the render component subcommand via the --tags argument.
This patch implements minimal rendering of a v1alpha4 platform using the
new render.Builder interface.
Tags aren't wired up yet, but this patch does cleanly separate Builder
interface from the Artifacts. Platform rendering doesn't have an
artifact itself, all artifacts are produced by rendering each component,
so we'll see how that works when we make the same changes to component
rendering, breaking it down to a render.Builder interface that sets
values in an Artifact.
The holos cli does not use an interface to handle different Platform api
versions. This makes it difficult to evolve the API in a backwards
compatible way.
This patch adds a top level switch statement to the `holos render
platform` command. The switch discriminates on the Platform API
version. v1alpha3 and earlier are classified as legacy versions and
will use the existing strict types. v1alpha4 and later versions will
use an interface to render the platform, allowing for multiple types to
implement the platform rendering interface.
PROBLEM:
The landing page contains a lot of text, and much of that text was
written before we refined our messaging within the guides and technical
overview pages.
SOLUTION:
* Whittle down landing page text to only the key messages we want to convey.
* Provide messaging bullets for the features.
* Steer folks (via links) to the quickstart guide or technical overview document.
OUTCOME:
Visitors don't need to wade through a lot of text to receive key
messaging talking points or links to the pages they should read.
PROBLEM:
There's a lot of text to grok on the landing page. A diagram would help
to visually convey what Holos does.
SOLUTION:
* Create a diagram
* Add to landing page
OUTCOME:
A visual aide is present on the landing page that helps explain where
Holos sits.
PROBLEM:
The Quickstart is lacking narrative tying the changes we're asking
people to make to the underlying organizational problems.
SOLUTION:
Improve the narrative to surface the problems we are solving and how
this affects the different teams at the Bank of Holos
OUTCOME:
Clarity on the problems the quickstart is solving.
Closes: #259
This gets us through to the end with podinfo deployed. Need to tell the
story of the migration team a bit better though, working with the
platform team to expose the service.
It's too long for documentation. Shorten it for clarity.
Result:
```
❯ holos render platform ./platform
rendered bank-accounts-db for cluster workload in 160.7245ms
rendered bank-ledger-db for cluster workload in 162.465625ms
rendered bank-userservice for cluster workload in 166.150417ms
rendered bank-ledger-writer for cluster workload in 168.075459ms
rendered bank-balance-reader for cluster workload in 172.492292ms
rendered bank-backend-config for cluster workload in 198.117916ms
rendered bank-secrets for cluster workload in 223.200042ms
rendered gateway for cluster workload in 124.841917ms
rendered httproutes for cluster workload in 131.86625ms
rendered bank-contacts for cluster workload in 154.463792ms
rendered bank-transaction-history for cluster workload in 159.968208ms
rendered bank-frontend for cluster workload in 325.24425ms
rendered app-projects for cluster workload in 110.577916ms
rendered ztunnel for cluster workload in 137.502792ms
rendered cni for cluster workload in 209.993375ms
rendered cert-manager for cluster workload in 172.933834ms
rendered external-secrets for cluster workload in 135.759792ms
rendered local-ca for cluster workload in 98.026708ms
rendered istiod for cluster workload in 403.050833ms
rendered argocd for cluster workload in 294.663167ms
rendered gateway-api for cluster workload in 228.47875ms
rendered namespaces for cluster workload in 113.586916ms
rendered base for cluster workload in 533.76675ms
rendered external-secrets-crds for cluster workload in 529.053375ms
rendered crds for cluster workload in 931.180458ms
rendered platform in 1.248310167s
```
Previously:
```
❯ holos render platform ./platform
rendered projects/bank-of-holos/backend/components/bank-ledger-db for cluster workload in 158.534875ms
rendered projects/bank-of-holos/backend/components/bank-accounts-db for cluster workload in 159.836166ms
rendered projects/bank-of-holos/backend/components/bank-userservice for cluster workload in 160.360667ms
rendered projects/bank-of-holos/backend/components/bank-balance-reader for cluster workload in 169.478584ms
rendered projects/bank-of-holos/backend/components/bank-ledger-writer for cluster workload in 169.437833ms
rendered projects/bank-of-holos/backend/components/bank-backend-config for cluster workload in 182.089333ms
rendered projects/bank-of-holos/security/components/bank-secrets for cluster workload in 196.502792ms
rendered projects/platform/components/istio/gateway for cluster workload in 122.273083ms
rendered projects/bank-of-holos/frontend/components/bank-frontend for cluster workload in 307.573584ms
rendered projects/platform/components/httproutes for cluster workload in 149.631583ms
rendered projects/bank-of-holos/backend/components/bank-contacts for cluster workload in 153.529708ms
rendered projects/bank-of-holos/backend/components/bank-transaction-history for cluster workload in 165.375667ms
rendered projects/platform/components/app-projects for cluster workload in 107.253958ms
rendered projects/platform/components/istio/ztunnel for cluster workload in 137.22275ms
rendered projects/platform/components/istio/cni for cluster workload in 233.980958ms
rendered projects/platform/components/cert-manager for cluster workload in 171.966958ms
rendered projects/platform/components/external-secrets for cluster workload in 134.207792ms
rendered projects/platform/components/istio/istiod for cluster workload in 403.19ms
rendered projects/platform/components/local-ca for cluster workload in 97.544708ms
rendered projects/platform/components/argocd/argocd for cluster workload in 289.577208ms
rendered projects/platform/components/gateway-api for cluster workload in 218.290458ms
rendered projects/platform/components/namespaces for cluster workload in 109.534125ms
rendered projects/platform/components/istio/base for cluster workload in 526.32525ms
rendered projects/platform/components/external-secrets-crds for cluster workload in 523.7495ms
rendered projects/platform/components/argocd/crds for cluster workload in 1.002546375s
rendered platform in 1.312824333s
```
Without this patch ArgoCD treats the Application as constantly out of
sync. This is also a good example of how to patch an arbitrary
component, though it patches the core BuildPlan itself now. If this is
widely used, it would be nice to add this behavior to the schema api
(aka author api).
Without this patch ArgoCD treats the Application as constantly out of
sync. This is also a good example of how to patch an arbitrary
component, though it patches the core BuildPlan itself now. If this is
widely used, it would be nice to add this behavior to the schema api
(aka author api).
Without this patch ArgoCD treats the Application as constantly out of
sync. This is also a good example of how to patch an arbitrary
component, though it patches the core BuildPlan itself now. If this is
widely used, it would be nice to add this behavior to the schema api
(aka author api).
Without this patch browsing https://bank.holos.localhost frequently gets
connection reset errors. These errors are caused by the frontend
deployment redirecting the browser to http, which is not enabled on the
Gateway we use in the guides.
This patch sets the scheme to https which corrects the problems.
See https://github.com/GoogleCloudPlatform/bank-of-anthos/issues/478
With this patch the frontend, accounts-db, and userservice all start and
become ready.
The user can log in, but on redirecting to home the site can't be
reached.
Rather than commit the jwt private key to version control like upstream
does, we use a SecretStore and ExternalSecret to sync the secret
generated by the security team in the bank-security namespace.
With this patch the SecretStore validates and the ExternalSecret
automatically syncs the secret from the bank-security namespace to the
bank-frontend namespace.
```
❯ k get ss
NAME AGE STATUS CAPABILITIES READY
bank-security 1s Valid ReadWrite True
❯ k get es
NAME STORE REFRESH INTERVAL STATUS READY
jwt-key bank-security 5s SecretSynced True
```
The pod start successfully.
```
❯ k get pods
NAME READY STATUS RESTARTS AGE
frontend-646d797d6b-7jhrx 1/1 Running 0 2m39s
❯ k logs frontend-646d797d6b-7jhrx
{"timestamp": "2024-09-16 21:44:47", "message": "info | Starting gunicorn 22.0.0", "severity": "INFO"}
{"timestamp": "2024-09-16 21:44:47", "message": "info | Listening at: http://0.0.0.0:8080 (7)", "severity": "INFO"}
{"timestamp": "2024-09-16 21:44:47", "message": "info | Using worker: gthread", "severity": "INFO"}
{"timestamp": "2024-09-16 21:44:47", "message": "info | Booting worker with pid: 8", "severity": "INFO"}
{"timestamp": "2024-09-16 21:44:57", "message": "create_app | Unable to retrieve cluster name from metadata server metadata.google.internal.", "severity": "WARNING"}
{"timestamp": "2024-09-16 21:44:57", "message": "create_app | Unable to retrieve zone from metadata server metadata.google.internal.", "severity": "WARNING"}
{"timestamp": "2024-09-16 21:44:57", "message": "create_app | Starting frontend service.", "severity": "INFO"}
{"timestamp": "2024-09-16 21:44:57", "message": "create_app | 🚫 Tracing disabled.", "severity": "INFO"}
{"timestamp": "2024-09-16 21:44:57", "message": "create_app | Platform is set to 'local'", "severity": "INFO"}
```
Expose Service frontend in the bank-frontend namespace via httproute
https://bank.holos.localhost
Organize into frontend, backend, security projects to align with three
teams who would each own this work.
remove secret from version control
Google added the secret to version control but we can generate the
secret in-cluster. Holos makes it easier to manage the ExternalSecret
or RoleBinding necessary to get it in the right place.
We need a way to demonstrate the value Holos offers in a platform team
managing projects for other teams. This patch addresses the need by
establishing the bank-of-holos schematic, which is a port of the Bank of
Anthos project to Holos.
This patch adds only the frontend to get the process started. As of
this patch the frontend pod starts and becomes ready but is not exposed
via HTTPRoute.
Refer to https://github.com/GoogleCloudPlatform/bank-of-anthos/
Previously all generated ArgoCD Application resources go into the
default project following the Quickstart guide. The configuration code
is being organized into the concept of projects in the filesystem, so we
want to the GitOps configuration to also reflect this concept of
projects.
This patch extends the ArgoConfig user facing schema to accept a project
string. The app-projects component automatically manages AppProject
resources in the argocd namespace for each of the defined projects.
This allows CUE configuration in the a project directory to specify the
project name so that all Applications are automatically assigned to the
correct project.
Providing ArgoConfig only works with the Helm kind without this patch.
This is a problem because we want to produce an Application for every
supported component kind when rendering the platform.
This patch threads the ArgoConfig struct described in the Quickstart
guide through every supported component kind.
Define a place for components to register HTTPRoute resources the
platform team needs to manage in the Gateway namespace.
The files are organized to delegate to the platform team.
This patch also fixes the naming of the argocd component so that the
Service is argocd-server instead of argo-cd-argocd-server.
Previously, the #Resources struct listing valid resources to use with
APIObjects in each of the components types was closed. This made it
very difficult for users to mix in new resources and use the Kubernetes
component kind.
This patch moves the definition of the valid resources to package holos
from the schema API. The schema still enforces some light constraints,
but doesn't keep the struct closed.
A new convention is introduced in the form of configuring all components
using _ComponentConfig defined at the root, then unifying this struct
with all of the component kinds. See schema.gen.cue for how this works.
This approach enables mixing in ArgoCD applications to all component
kinds, not just Helm as was done previously. Similarly, the
user-constrained #Resources definition unifies with all component kinds.
It's OK to leave the yaml.Marshall in the schema API. The user
shouldn't ever have to deal with #APIObjects, instead they should pass
Resources through the schema API which will use APIObjects to create
apiObjectMap for each component type and the BuildPlan.
This is still more awkward than I want, but it's a good step in the
right direction.
Without this patch the istio-gateway component isn't functional, the
HTTPRoute created for httpbin isn't programmed correctly. There is no
Gateway resource, just a deployment created by the istio helm chart.
This patch replaces the helm chart with a Gateway resource as was done
previously in the k3d platform schematic.
This patch also simplifies the certificate management to issue a single
cert valid for the platform domain and a wildcard. We intentionally
avoid building a dynamic Gateway.spec.listeners structure to keep the
expose a service guide relatively simple and focused on getting started
with Holos.
This patch adds the httpbin routes component. It's missing the
Certificate component, the next step is to wire up automatic certificate
management in the gateway configuration, which is a prime use case for
holos. Similar to how we register components and namespaces, we'll
register certificates.
This patch also adds the #Platform.Domain field to the user facing
schema API. We previously stored the domain in the Model but it makes
sense to lift it up to the Platform and have a sensible default value
for it.
Another example of #237 needing to be addressed soon.
This patch manages the httpbin Deployment, Service, and ReferenceGrant.
The remaining final step is to expose the service with an HTTPRoute and
Certificate.
We again needed to add a field to the schema APIObjects to get this to
work. We need to fix#237 soon. We'll need to do it again for the
HTTPRoute and Certificate resources.
The progression of namespaces, cert-manager, then gateway api and istio
makes much more sense than the previous progression of gateway api,
namespaces, istio.
cert-manager builds nicely on top of namespaces. gateway api are only
crds necessary for istio.
This patch also adds the local-ca component which surfaces issue #237
The Kubernetes APIObjects are unnecessarily constrained to resources we
define in the schema. We need to move the marshal code into package
holos so the user can add their own resource kinds.
This patch adds Istio to the Expose a Service documentation and
introduces new concepts. The Kubernetes build plan schema, the
namespaces component, and an example of how to safely re-use Helm values
from the root to multiple leaf components.
fix: istio cni not ready on k3d
---
The istio-k3d component embedded into holos fixes the cni pod not
becoming ready with our k3d local cluster guide. The pod log error this
fixes is:
configuration requires updates, (re)writing CNI config file at "": no networks found in /host/etc/cni/net.d
Istio CNI is configured as chained plugin, but cannot find existing CNI network config: no networks found in /host/etc/cni/net.d
Waiting for CNI network config file to be written in /host/etc/cni/net.d...
[Platform k3d]: https://istio.io/latest/docs/ambient/install/platform-prerequisites/#k3d
docs: clarify how to reset the local cluster
---
This is something we do all the time while developing and documenting,
so make it easy and fast to reset the cluster to a known good state.
This patch adds the schema api for the Kubernetes build plan, which
produces plain API resources directly from CUE. It's needed for the
namespaces component which is foundational to many of our guides.
The first guide that needs this is the expose a service guide, we need
to register the namespaces from the istio component.
The Expose a Service doc is meant to be the second step after the
Quickstart doc. This commit adds the section describing how to install
the Gateway API.
The Kustomize build plan is introduced at this point in a similar way
the Helm build plan was introduced in the quickstart.
We need an easy way to help people add a workload cluster to their
workload fleet when working through the guides. Generated platforms
should not define any clusters so they can be reused with multiple
guides.
This patch adds a simple component schematic that drops a root cue file
to define a workload cluster named workload.
The result is the following sequence renders the Gateway API when run
from an empty directory.
holos generate platform guide
holos generate component workload-cluster
holos generate component gateway-api
holos render platform ./platform
Without this patch nothing is rendered because there are no workload
clusters in the base guide platform.
Having the management cluster hard coded into the definition of the
standard fleets is problematic for guides that don't need a management
cluster.
Define the fleets, but leave the set of clusters empty until they're
needed.
Previously helm and cue components were split into two different
subcommands off the holos generate component command. This is
unnecessary, I'm not sure why it was there in the first place. The code
seemed perfectly duplicated.
This patch combines them to focus on the concept of a Component. It
doesn't matter what kind it is now that it's expected to be run from the
root of the platform repository and drop configuration at the root and
the leaf of the tree.
Previously, each document needed to be manually included in the sidebars
to show up. In addition, index paths like /docs/ and /docs/guides/ were
not found.
This patch addresses both problems by switching sidebars to
automatically generate from filesystem directories. Important documents
like the getting started guide and introduction are expected to add a
`slug: /foo` front matter item to create a permalink.
The result is the sidebar reflects the filesystem while the URL bar is
more of a permalink. Files should be able to be moved around the file
system and the sidebar tree without affecting their URL.
This patch also consolidates the API and Docs sidebars into one.
Our guides should be useful reading them only from a mobile device. For
those readers who also want to apply the manifests to a real cluster we
need a companion guide that describes how to get one.
This patch adds that guide, adapted from the old try holos locally page.
This patch incorporates the main feedback from Gary and Nate from this
morning. The note tab in argocd.cue was awkware to Gary and I. The use
of _ in CUE needs an explicit comment which this patch adds.
This patch focuses on the Day 2 benefits holos offers, specifically
making it easier to visiualize exactly what will change when upgrading
components.
In addition, it's easier to apply changes slowly and deliberately since
they're all just flat files in the local filesystem and Git repository.
Previously the quickstart didn't cover adding workload clusters and
rendering a platform with multiple clusters. This patch demonstrates
how it's effectively a one line change to clone the configuration of a
workload cluster to another geographic region.
Make sure go install works from the quickstart documentation by doing a
release. Otherwise, v0.93.1 is installed which doesn't include the
platform schema.
Previously, the quickstart step of generating the pod info component and
generating the platform as a whole left the task of integrating the
Component into the Platform as an exercise for the reader. This is a
problem because it creates unnecessary friction.
This patch addresses the problem by lifting up the Platform concept
into the user-facing Schema API. The generated platform includes a top
level #Platform definition which exposes the core Platform specification
on the Output field.
The Platform CUE instance then reduces to a simple `#Platform.Output`
which provides the Platform spec to holos for rendering each component
for each cluster.
The CUE code for the schema.#Platform iterates over each
Component to derive the list of components to manage for the Platform.
The CUE code for the generated quickstart platform links the definition
of StandardFleets, which is a Workload fleet and a Management cluster
fleet to the Platform conveninece wrapper.
Finally, the generated podinfo component drops a CUE file at the
repository root to automatically add the component to every workload
cluster.
The result is the only task left for the end user is to define at least
one workload cluster. Once defined, the component is automatically
managed because it is managed on all workload clusters.
This approach futher opens the door to allow generated components to
define their namespaces and generated secrets on the management cluster
separate from their workloads on the workload clusters.
This patch includes a behavior change, from now on all generated
components should assume they are writing to the root of the user's Git
repository so that they can generate files through the whole tree.
In the future, we should template output paths for generated components.
A simple approach might be to embed a file with a .target suffix, with
the contents being a simple Go template of the file path to write to.
The holos generate subcommand can then check if any given embedded file
foo has a foo.target companion, then write the target to the rendered
template value.
Users need to customize the default behavior of the core components,
like the Helm schema wrapper to mix-in an ArgoCD Application resource to
each component. This patch wires up #Helm in the holos package to
schema.#Helm from the v1alpha3 api.
The result is illustrated in the Quickstart documentation, it is now
simple for users to modify the definition of a Helm component such that
Application resources are mixed in to every component in the platform.
Previosly the end user needed to write, or at least copy and paste, a
large amount of boiler plate code to achieve the goal of declaring a
helm chart component. There is a gap between the cue code:
(#Helm & Chart).Output
And the full BuildPlan produced for the Holos cli to execute the
rendering process. The boiler plate code in schema.cue at the root of
the platform infrastructure repository was largely responsible for
defining how a BuildPlan with one HelmChart component is derived from
this #Helm definition.
This patch moves the definitions into a new, documented API named
`schema`. End users are expected to define their own #Helm definition
using the schema.#Helm, like so in the root level schema.cue:
#Helm: schema.#Helm
Without this patch deployments to the dev environment are failing with
the following error when commits are pushed to the main branch.
GIT_DETAIL=v0.93.0-3-g4db3fb4 GIT_SUFFIX= bash ./hack/deploy-dev
Cloning into 'holos-infra'...
could not validate
could not run: could not validate invalid BuildPlan: apiVersion invalid: want: v1alpha3 have: v1alpha2 at internal/builder/builder.go:308
could not run: could not render component: exit status 1 at internal/render/platform.go:48
make: *** [Makefile:147: dev-deploy] Error 1
This patch removes the api version check in the build plan validation
function. In the future, we should pass an interface internally in the
holos executable.
The result is holos render platform ./platform succeeds with this patch
applied.
Previously the CUE code needed to specify the Platform.spec.model field,
which created friction. This patch adds a cue struct tag to unify the
field with an open struct.
❯ holos render platform ./platform --log-level=debug
could not run: could not marshal cue instance platform: cue: marshal error: spec.model: cannot convert incomplete value "_" to JSON at internal/builder/platform.go:45
spec.model: cannot convert incomplete value "_" to JSON
The render command completes successfully with this patch without the
user having to provide a value for the spec.model field.
This patch adds the minimal amount of CUE code necessary to successfully
run the following two commands from the quickstart.
holos generate platform quickstart
holos render platform ./platform
The result is no componets are rendered, so nothing is done, but it does
succeeed.
This patch surfaces some friction and inconsistency with how the Model
is passed in and the initial structure of the _PlatformConfig. The tags
are required otherwise holos errors out.
Without this patch the `holos generate platform` command automatically
makes an rpc call to holos server. This creates friction for the
quickstart guide because we don't need to require users to register and
have an organization and platform already created in the server just to
generate a simple platform to exercise a simple helm chart component.
A future patch should implement the behavior of linking a server side
platform to a local git repository by making the API call to get the
platform ID then updating the platform.metadata.json file.
Previously the landing page focused on Holos as a reference platform.
We're refocusing the release on the holos package management tool. This
patch updates the landing page and adds placeholders for a new quick
start guide which will focus on wrapping a helm chart and a concepts
page which will provide a high level overview of how holos is unique
from other tools.
In an effort to increase reliability when trying holos locally. The
idea being generate to render platform should ideally work without a
network connection provided the executable has already been downloaded.
For example, to give a quick demo without a network connection.
Without this patch the argo install manifest may fail because the
resources are fetched from github.
This patch embeds the same resources to increase speed and reliability.
Without this patch the argo crds component takes a few seconds to render
and may fail because the resources are fetched from github.
This patch embeds the same resources to increase speed and reliability.
Without this patch the gateway api component takes a few seconds to
render and may fail because the resources are fetched from github.
This patch embeds the same resources to increase speed and reliability.
Result:
rendered components/gateway-api for cluster workload in 257.206208ms
Building the cluster today I got hung up on a `ERR_CONNECTION_CLOSED`
error from Chrome when trying to access httpbin.
The problem was I forgot to run the local-ca script, thinking I already
had a local ca. The problem is the script also copies the private key
to the cluster, so it must be run every time the cluster is created.
This patch clarifies the sequence. When resetting, everything following
the Create the Cluster step needs to be executed.
Previously the image is build on merge to main, but not deployed
anywhere. This patch adds steps to the publish workflow to deploy the
image that was published using gitops and argocd.
could not run: holos.spec.artifacts.0.transformers.0.kustomize.kustomization.patches.0.target.name: cannot convert non-concrete value string at builder/v1alpha5/builder.go:218
holos.spec.artifacts.0.transformers.0.kustomize.kustomization.patches.0.target.name: cannot convert non-concrete value string:
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Holos is a configuration management tool for Kubernetes that implements the rendered manifests pattern using CUE. It unifies Helm charts, Kustomize bases, and raw Kubernetes manifests into a single, declarative pipeline.
-`/api/core/v1alpha*/types.go` - API type definitions
### Component Types
1.**Helm** - Wraps Helm charts
2.**Kustomize** - Wraps Kustomize bases
3.**Kubernetes** - Raw Kubernetes manifests
## CUE Patterns
Components are defined in CUE:
```cue
packageholos
holos:Component.BuildPlan
Component:#Helm&{
Name:"example"
Chart:{
version:"1.0.0"
repository:{
name:"example"
url:"https://charts.example.com"
}
}
}
```
## Testing
- Unit tests: `*_test.go` files colocated with source
- Integration tests: `/cmd/holos/tests/`
- Example platforms: `/internal/testutil/fixtures/`
- Run single test: `go test -run TestName ./path/to/package`
## Development Patterns
1. Error handling: Prefer `errors.Format()` from `/internal/errors/` over `fmt.Errorf()`
2. Logging: Use structured `slog`, get logger with `logger.FromContext(ctx)`
3. CLI commands: Follow Cobra patterns in `/internal/cli/`
4. CUE formatting: Always run `cue fmt` on CUE files
5. Go formatting: Always run `go fmt` on go files
6. Develop against v1alpha6 packages.
7. Commits: Use the package name as the first word in the commit, lower case. Commit without asking permission. Always run `make lint` and `make test` before committing.
## Version Management
- Version files: `/version/embedded/{major,minor,patch}`
- Bump version: `make bump`
- API versions: v1alpha5 (stable), v1alpha6 (development)
## Key Concepts
- **Platform**: Top-level configuration containing all components
- **Component**: Unit of configuration (DAG of Tasks producing deployment configs for one component)
- **TaskSet**: DAG of Tasks (Similar to how make tasks behave)
- **BuildPlan**: Instructions for building a component. Deprecated in v1alpha6, use TaskSet instead.
- **Generator**: Creates manifests (Helm, Kustomize, etc.) author schema only in v1alpha6
- **Transformer**: Modifies generated manifests, author schema only in v1alpha6
- **Validator**: Validates final manifests, author schema only in v1alpha6
// Label is an arbitrary unique identifier internal to holos itself. The holos
// cli is expected to never write a Label value to rendered output files,
// therefore use a [Label] then the identifier must be unique and internal.
// Defined as a type for clarity and type checking.
//
// A Label is useful to convert a CUE struct to a list, for example producing a list of [APIObject] resources from an [APIObjectMap]. A CUE struct using
// Label keys is guaranteed to not lose data when rendering output because a
// Label is expected to never be written to the final output.
typeLabelstring
// Kind is a kubernetes api object kind. Defined as a type for clarity and type checking.
typeKindstring
// APIObject represents the most basic generic form of a single kubernetes api
// object. Represented as a JSON object internally for compatibility between
// tools, for example loading from CUE.
typeAPIObjectstructpb.Struct
// APIObjectMap represents the marshalled yaml representation of kubernetes api
// objects. Do not produce an APIObjectMap directly, instead use [APIObjects]
// to produce the marshalled yaml representation from CUE data, then provide the
// result to [HolosComponent].
typeAPIObjectMapmap[Kind]map[Label]string
// APIObjects represents Kubernetes API objects defined directly from CUE code.
// Useful to mix in resources to any kind of [HolosComponent], for example
// adding an ExternalSecret resource to a [HelmChart].
//
// [Kind] must be the resource kind, e.g. Deployment or Service.
//
// [Label] is an arbitrary internal identifier to uniquely identify the resource
// within the context of a `holos` command. Holos will never write the
// intermediate label to rendered output.
//
// Refer to [HolosComponent] which accepts an [APIObjectMap] field provided by
stderr 'Error: execution error at \(zitadel/templates/secret_zitadel-masterkey.yaml:2:4\): Either set .Values.zitadel.masterkey xor .Values.zitadel.masterkeySecretName'
# Reference the name of a secret that contains ZITADEL configuration.
configSecretName:
# The key under which the ZITADEL configuration is located in the secret.
configSecretKey: config-yaml
# ZITADEL uses the masterkey for symmetric encryption.
# You can generate it for example with tr -dc A-Za-z0-9 </dev/urandom | head -c 32
masterkey: ""
# Reference the name of the secret that contains the masterkey. The key should be named "masterkey".
# Note: Either zitadel.masterkey or zitadel.masterkeySecretName must be set
masterkeySecretName: ""
# Annotations set on masterkey secret
masterkeyAnnotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation
helm.sh/hook-weight: "0"
# The CA Certificate needed for establishing secure database connections
dbSslCaCrt: ""
# The Secret containing the CA certificate at key ca.crt needed for establishing secure database connections
dbSslCaCrtSecret: ""
# The db admins secret containing the client certificate and key at tls.crt and tls.key needed for establishing secure database connections
dbSslAdminCrtSecret: ""
# The db users secret containing the client certificate and key at tls.crt and tls.key needed for establishing secure database connections
dbSslUserCrtSecret: ""
# Generate a self-signed certificate using an init container
# This will also mount the generated files to /etc/tls/ so that you can reference them in the pod.
# E.G. KeyPath: /etc/tls/tls.key CertPath: /etc/tls/tls.crt
# By default, the SAN DNS names include, localhost, the POD IP address and the POD name. You may include one more by using additionalDnsName like "my.zitadel.fqdn".
selfSignedCert:
enabled: false
additionalDnsName:
replicaCount: 3
image:
repository: ghcr.io/zitadel/zitadel
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
chownImage:
repository: alpine
pullPolicy: IfNotPresent
tag: "3.19"
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
# Annotations to add to the deployment
annotations: {}
# Annotations to add to the configMap
configMap:
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation
helm.sh/hook-weight: "0"
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation
helm.sh/hook-weight: "0"
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
podAnnotations: {}
podAdditionalLabels: {}
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
securityContext: {}
# Additional environment variables
env:
[]
# - name: ZITADEL_DATABASE_POSTGRES_HOST
# valueFrom:
# secretKeyRef:
# name: postgres-pguser-postgres
# key: host
service:
type: ClusterIP
# If service type is "ClusterIP", this can optionally be set to a fixed IP address.
clusterIP: ""
port: 8080
protocol: http2
annotations: {}
scheme: HTTP
ingress:
enabled: false
className: ""
annotations: {}
hosts:
- host: localhost
paths:
- path: /
pathType: Prefix
tls: []
resources: {}
nodeSelector: {}
tolerations: []
affinity: {}
topologySpreadConstraints: []
initJob:
# Once ZITADEL is installed, the initJob can be disabled.
Package author contains a standard set of schemas for component authors to generate common [core](<https://holos.run/docs/api/core/>) BuildPlans.
Holos values stability, flexibility, and composition. This package intentionally defines only the minimal necessary set of structures. Component authors are encouraged to define their own structures building on our example [topics](<https://holos.run/docs/topics/>).
The Holos Maintainers may add definitions to this package if the community identifies nearly all users must define the exact same structure. Otherwise, definitions should be added as a customizable example in [topics](<https://holos.run/docs/topics/>).
For example, structures representing a cluster and environment almost always need to be defined. Their definition varies from one organization to the next. Therefore, customizable definitions for a cluster and environment are best maintained in [topics](<https://holos.run/docs/topics/>), not standardized in this package.
## Index
- [type ComponentConfig](<#ComponentConfig>)
- [type Helm](<#Helm>)
- [type Kubernetes](<#Kubernetes>)
- [type Kustomize](<#Kustomize>)
- [type KustomizeConfig](<#KustomizeConfig>)
- [type NameLabel](<#NameLabel>)
- [type Platform](<#Platform>)
<a name="ComponentConfig"></a>
## type ComponentConfig {#ComponentConfig}
ComponentConfig represents the configuration common to all kinds of components for use with the holos render component command. All component kinds may be transformed with [kustomize](<https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/>) configured with the [KustomizeConfig](<#KustomizeConfig>) field.
- [Helm](<#Helm>) charts.
- [Kubernetes](<#Kubernetes>) resources generated from CUE.
- [Kustomize](<#Kustomize>) bases.
```go
typeComponentConfigstruct{
// Name represents the BuildPlan metadata.name field. Used to construct the
// fully rendered manifest file path.
Namestring
// Labels represent the BuildPlan metadata.labels field.
Labelsmap[string]string
// Annotations represent the BuildPlan metadata.annotations field.
Annotationsmap[string]string
// Path represents the path to the component producing the BuildPlan.
Pathstring
// Parameters are useful to reuse a component with various parameters.
// Injected as CUE @tag variables. Parameters with a "holos_" prefix are
// reserved for use by the Holos Authors.
Parametersmap[string]string
// OutputBaseDir represents the output base directory used when assembling
// artifacts. Useful to organize components by clusters or other parameters.
// For example, holos writes resource manifests to
// Resources represents kubernetes resources mixed into the rendered manifest.
Resourcescore.Resources
// KustomizeConfig represents the kustomize configuration.
KustomizeConfigKustomizeConfig
// Validators represent checks that must pass for output to be written.
Validatorsmap[NameLabel]core.Validator
// Artifacts represents additional artifacts to mix in. Useful for adding
// GitOps resources. Each Artifact is unified without modification into the
// BuildPlan.
Artifactsmap[NameLabel]core.Artifact
}
```
<a name="Helm"></a>
## type Helm {#Helm}
Helm assembles a BuildPlan rendering a helm chart. Useful to mix in additional resources from CUE and transform the helm output with kustomize.
```go
typeHelmstruct{
ComponentConfig`json:",inline"`
// Chart represents a Helm chart.
Chartcore.Chart
// Values represents data to marshal into a values.yaml for helm.
Valuescore.Values
// ValueFiles represents value files for migration from helm value
// hierarchies. Use Values instead.
ValueFiles[]core.ValueFile`json:",omitempty"`
// EnableHooks enables helm hooks when executing the `helm template` command.
EnableHooksbool`cue:"true | *false"`
// Namespace sets the helm chart namespace flag if provided.
Namespacestring`json:",omitempty"`
// APIVersions represents the helm template --api-versions flag
APIVersions[]string`json:",omitempty"`
// KubeVersion represents the helm template --kube-version flag
KubeVersionstring`json:",omitempty"`
// BuildPlan represents the derived BuildPlan produced for the holos render
// component command.
BuildPlancore.BuildPlan
}
```
<a name="Kubernetes"></a>
## type Kubernetes {#Kubernetes}
Kubernetes assembles a BuildPlan containing inline resources exported from CUE.
```go
typeKubernetesstruct{
ComponentConfig`json:",inline"`
// BuildPlan represents the derived BuildPlan produced for the holos render
// component command.
BuildPlancore.BuildPlan
}
```
<a name="Kustomize"></a>
## type Kustomize {#Kustomize}
Kustomize assembles a BuildPlan rendering manifests from a [kustomize](<https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/>) kustomization.
```go
typeKustomizestruct{
ComponentConfig`json:",inline"`
// BuildPlan represents the derived BuildPlan produced for the holos render
// component command.
BuildPlancore.BuildPlan
}
```
<a name="KustomizeConfig"></a>
## type KustomizeConfig {#KustomizeConfig}
KustomizeConfig represents the configuration for [kustomize](<https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/>) post processing. Use the Files field to mix in plain manifest files located in the component directory. Use the Resources field to mix in manifests from network urls.
```go
typeKustomizeConfigstruct{
// Kustomization represents the kustomization used to transform resources.
// Note the resources field is internally managed from the Files and Resources fields.
Kustomizationmap[string]any`json:",omitempty"`
// Files represents files to copy from the component directory for kustomization.
// CommonLabels represents common labels added without including selectors.
CommonLabelsmap[string]string
}
```
<a name="NameLabel"></a>
## type NameLabel {#NameLabel}
NameLabel represents the common use case of converting a struct to a list where the name field of each value unifies with the field name of the outer struct.
For example:
```
S: [NameLabel=string]: name: NameLabel
S: jeff: _
S: gary: _
S: nate: _
L: [for x in S {x}]
// L is [{name: "jeff"}, {name: "gary"}, {name: "nate"}]
```
```go
typeNameLabelstring
```
<a name="Platform"></a>
## type Platform {#Platform}
Platform assembles a core Platform in the Resource field for the holos render platform command. Use the Components field to register components with the platform.
Package core contains schemas for a [Platform](<#Platform>) and [BuildPlan](<#BuildPlan>). Holos takes a [Platform](<#Platform>) as input, then iterates over each [Component](<#Component>) to produce a [BuildPlan](<#BuildPlan>). Holos processes the [BuildPlan](<#BuildPlan>) to produce fully rendered manifests, each an [Artifact](<#Artifact>).
<a name="BuildContextTag"></a>BuildContextTag represents the cue tag holos render component uses to inject the json representation of a [BuildContext](<#BuildContext>) for use in a BuildPlan.
```go
constBuildContextTagstring="holos_build_context"
```
<a name="ComponentAnnotationsTag"></a>ComponentAnnotationsTag represents the tag holos uses to inject the json representation of [Component](<#Component>) metadata annotations from the holos render platform command to the holos render component command.
<a name="ComponentLabelsTag"></a>ComponentLabelsTag represents the cue tag holos uses to inject the json representation of [Component](<#Component>) metadata labels from the holos render platform command to the holos render component command.
<a name="ComponentNameTag"></a>ComponentNameTag represents the cue tag holos uses to inject a [Component](<#Component>) name from the holos render platform command to the holos render component command.
<a name="ComponentPathTag"></a>ComponentPathTag represents the cue tag holos uses to inject a [Component](<#Component>) path relative to the cue module root from the holos render platform command to the holos render component command.
Artifact represents one fully rendered manifest produced by a [Transformer](<#Transformer>) sequence, which transforms a [Generator](<#Generator>) collection. A [BuildPlan](<#BuildPlan>) produces an [Artifact](<#Artifact>) collection.
Each Artifact produces one manifest file or directory artifact. Generator Output values are used as Transformer Inputs. The Output field of the final [Transformer](<#Transformer>) should have the same value as the Artifact field.
When there is more than one [Generator](<#Generator>) there should be at least one [Transformer](<#Transformer>) to combine outputs into one Artifact file, or the final artifact should be a directory containing the outputs of the generators. If there is a single Generator, it may directly produce the Artifact output.
An Artifact is processed concurrently with other artifacts in the same [BuildPlan](<#BuildPlan>). One Artifact must not use an output of another Artifact as an input. Each [Generator](<#Generator>) within an artifact also runs concurrently with generators of the same artifact. Each [Transformer](<#Transformer>) is executed sequentially starting after all generators have completed.
Output fields are write\-once. It is an error for multiple Generators or Transformers to produce the same Output value within the context of a [BuildPlan](<#BuildPlan>).
When directories are used as inputs or outputs, they behave similar to how \`git\` works with directories. When the output field references a directory, all files within the directory are recursively stored using their relative path as a key. Similar to git add . When the input field references an absent file, a / is appended and the resulting value is used as a prefix match against all previous task outputs.
BuildContext represents build context values owned by the holos render component command. End users should not manage context field values. End users may reference fields from within CUE to refer to late binding concrete values defined just before holos executes a [BuildPlan](<#BuildPlan>).
Holos injects build context values by marshalling this struct to json through the holos\_build\_context cue tag.
Example usage from cue to produce a [BuildPlan](<#BuildPlan>):
BuildPlan represents an implementation of the [rendered manifest pattern](<https://akuity.io/blog/the-rendered-manifests-pattern>). Holos processes a BuildPlan to produce one or more [Artifact](<#Artifact>) output files. BuildPlan artifact files usually contain Kubernetes manifests, but they may have any content.
A BuildPlan usually produces two artifacts. One artifact contains a manifest of resources. A second artifact contains a GitOps resource to manage the first, usually an ArgoCD Application resource.
Holos uses CUE to construct a BuildPlan. Holos injects late binding values such as the build temp dir using the [BuildContext](<#BuildContext>).
```go
typeBuildPlanstruct{
// APIVersion represents the versioned schema of the resource.
Command represents a [BuildPlan](<#BuildPlan>) task implemented by executing an user defined system command. A task is defined as a [Generator](<#Generator>), [Transformer](<#Transformer>), or [Validator](<#Validator>). Commands are executed with the working directory set to the platform root.
```go
typeCommandstruct{
// DisplayName of the command. The basename of args[0] is used if empty.
File represents a simple single file copy [Generator](<#Generator>). Useful with a [Kustomize](<#Kustomize>) [Transformer](<#Transformer>) to process plain manifest files stored in the component directory. Multiple File generators may be used to transform multiple resources.
```go
typeFilestruct{
// Source represents a file sub-path relative to the component path.
SourceFilePath`json:"source" yaml:"source"`
}
```
<a name="FileContent"></a>
## type FileContent {#FileContent}
FileContent represents file contents.
```go
typeFileContentstring
```
<a name="FileContentMap"></a>
## type FileContentMap {#FileContentMap}
FileContentMap represents a mapping of file paths to file contents.
```go
typeFileContentMapmap[FilePath]FileContent
```
<a name="FileOrDirectoryPath"></a>
## type FileOrDirectoryPath {#FileOrDirectoryPath}
FileOrDirectoryPath represents a file or a directory path.
```go
typeFileOrDirectoryPathstring
```
<a name="FilePath"></a>
## type FilePath {#FilePath}
FilePath represents a file path.
```go
typeFilePathstring
```
<a name="Generator"></a>
## type Generator {#Generator}
Generator generates Kubernetes resources. [Helm](<#Helm>) and [Resources](<#Resources>) are the most commonly used, often paired together to mix\-in resources to an unmodified Helm chart. A simple [File](<#File>) generator is also available for use with the [Kustomize](<#Kustomize>) transformer.
Each Generator in an [Artifact](<#Artifact>) must have a distinct Output value for a [Transformer](<#Transformer>) to reference.
1. [Resources](<#Resources>) \- Generates resources from CUE code.
2. [Helm](<#Helm>) \- Generates rendered yaml from a [Chart](<#Chart>).
3. [File](<#File>) \- Generates data by reading a file from the component directory.
4. [Command](<#Command>) \- Generates data by executing an user defined command.
```go
typeGeneratorstruct{
// Kind represents the kind of generator. Must be Resources, Helm, or File.
InternalLabel is an arbitrary unique identifier internal to holos itself. The holos cli is expected to never write a InternalLabel value to rendered output files, therefore use a InternalLabel when the identifier must be unique and internal. Defined as a type for clarity and type checking.
```go
typeInternalLabelstring
```
<a name="Join"></a>
## type Join {#Join}
Join represents a [Transformer](<#Transformer>) using [bytes.Join](<https://pkg.go.dev/bytes#Join>) to concatenate multiple inputs into one output with a separator. Useful for combining output from [Helm](<#Helm>) and [Resources](<#Resources>) together into one [Artifact](<#Artifact>) when [Kustomize](<#Kustomize>) is otherwise unnecessary.
Kind is a discriminator. Defined as a type for clarity and type checking.
```go
typeKindstring
```
<a name="Kustomization"></a>
## type Kustomization {#Kustomization}
Kustomization represents a kustomization.yaml file for use with the [Kustomize](<#Kustomize>) [Transformer](<#Transformer>). Untyped to avoid tightly coupling holos to kubectl versions which was a problem for the Flux maintainers. Type checking is expected to happen in CUE against the kubectl version the user prefers.
```go
typeKustomizationmap[string]any
```
<a name="Kustomize"></a>
## type Kustomize {#Kustomize}
Kustomize represents a kustomization [Transformer](<#Transformer>).
```go
typeKustomizestruct{
// Kustomization represents the decoded kustomization.yaml file
Platform represents a platform to manage. A Platform specifies a [Component](<#Component>) collection and integrates the components together into a holistic platform. Holos iterates over the [Component](<#Component>) collection producing a [BuildPlan](<#BuildPlan>) for each, which holos then executes to render manifests.
Inspect a Platform resource holos would process by executing:
```
cue export --out yaml ./platform
```
```go
typePlatformstruct{
// APIVersion represents the versioned schema of this resource.
Repository represents a [Helm](<#Helm>) [Chart](<#Chart>) repository.
The Auth field is useful to configure http basic authentication to the Helm repository. Holos gets the username and password from the environment variables represented by the Auth field.
Resources represents Kubernetes resources. Most commonly used to mix resources into the [BuildPlan](<#BuildPlan>) generated from CUE, but may be generated from elsewhere.
```go
typeResourcesmap[Kind]map[InternalLabel]Resource
```
<a name="Transformer"></a>
## type Transformer {#Transformer}
Transformer combines multiple inputs from prior [Generator](<#Generator>) or [Transformer](<#Transformer>) outputs into one output. [Kustomize](<#Kustomize>) is the most commonly used transformer. A simple [Join](<#Join>) is also supported for use with plain manifest files.
1. [Kustomize](<#Kustomize>) \- Patch and transform the output from prior generators or transformers. See [Introduction to Kustomize](<https://kubectl.docs.kubernetes.io/guides/config_management/introduction/>).
2. [Join](<#Join>) \- Concatenate multiple prior outputs into one output.
3. [Command](<#Command>) \- Transforms data by executing an user defined command.
```go
typeTransformerstruct{
// Kind represents the kind of transformer. Must be Kustomize, or Join.
Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Holos may execute validators concurrently. See the [validators](<https://holos.run/docs/v1alpha6/tutorial/validators/>) tutorial for an end to end example.
```go
typeValidatorstruct{
// Kind represents the kind of transformer. Must be Kustomize, or Join.
Package v1alpha2 contains the core API contract between the holos cli and CUE configuration code. Platform designers, operators, and software developers use this API to write configuration in CUE which \`holos\` loads. The overall shape of the API defines imperative actions \`holos\` should carry out to render the complete yaml that represents a Platform.
[Platform](<#Platform>) defines the complete configuration of a platform. With the holos reference platform this takes the shape of one management cluster and at least two workload cluster. Each cluster has multiple [HolosComponent](<#HolosComponent>) resources applied to it.
Each holos component path, e.g. \`components/namespaces\` produces exactly one [BuildPlan](<#BuildPlan>) which in turn contains a set of [HolosComponent](<#HolosComponent>) kinds.
The primary kinds of [HolosComponent](<#HolosComponent>) are:
1. [HelmChart](<#HelmChart>) to render config from a helm chart.
2. [KustomizeBuild](<#KustomizeBuild>) to render config from [Kustomize](<#Kustomize>)
3. [KubernetesObjects](<#KubernetesObjects>) to render [APIObjects](<#APIObjects>) defined directly in CUE configuration.
Note that Holos operates as a data pipeline, so the output of a [HelmChart](<#HelmChart>) may be provided to [Kustomize](<#Kustomize>) for post\-processing.
// ChartDir is the directory name created in the holos component directory to cache a chart.
ChartDir="vendor"
// ResourcesFile is the file name used to store component output when post-processing with kustomize.
ResourcesFile="resources.yaml"
)
```
<a name="KubernetesObjectsKind"></a>
```go
constKubernetesObjectsKind="KubernetesObjects"
```
<a name="APIObject"></a>
## type APIObject {#APIObject}
APIObject represents the most basic generic form of a single kubernetes api object. Represented as a JSON object internally for compatibility between tools, for example loading from CUE.
```go
typeAPIObjectstructpb.Struct
```
<a name="APIObjectMap"></a>
## type APIObjectMap {#APIObjectMap}
APIObjectMap represents the marshalled yaml representation of kubernetes api objects. Do not produce an APIObjectMap directly, instead use [APIObjects](<#APIObjects>) to produce the marshalled yaml representation from CUE data, then provide the result to [HolosComponent](<#HolosComponent>).
```go
typeAPIObjectMapmap[Kind]map[Label]string
```
<a name="APIObjects"></a>
## type APIObjects {#APIObjects}
APIObjects represents Kubernetes API objects defined directly from CUE code. Useful to mix in resources to any kind of [HolosComponent](<#HolosComponent>), for example adding an ExternalSecret resource to a [HelmChart](<#HelmChart>).
[Kind](<#Kind>) must be the resource kind, e.g. Deployment or Service.
[Label](<#Label>) is an arbitrary internal identifier to uniquely identify the resource within the context of a \`holos\` command. Holos will never write the intermediate label to rendered output.
Refer to [HolosComponent](<#HolosComponent>) which accepts an [APIObjectMap](<#APIObjectMap>) field provided by [APIObjects](<#APIObjects>).
BuildPlan represents a build plan for the holos cli to execute. The purpose of a BuildPlan is to define one or more [HolosComponent](<#HolosComponent>) kinds. For example a [HelmChart](<#HelmChart>), [KustomizeBuild](<#KustomizeBuild>), or [KubernetesObjects](<#KubernetesObjects>).
A BuildPlan usually has an additional empty [KubernetesObjects](<#KubernetesObjects>) for the purpose of using the [HolosComponent](<#HolosComponent>) DeployFiles field to deploy an ArgoCD or Flux gitops resource for the holos component.
// Release represents the chart release when executing helm template.
Releasestring`json:"release"`
// Repository represents the repository to fetch the chart from.
RepositoryRepository`json:"repository,omitempty"`
}
```
<a name="FileContent"></a>
## type FileContent {#FileContent}
FileContent represents file contents.
```go
typeFileContentstring
```
<a name="FileContentMap"></a>
## type FileContentMap {#FileContentMap}
FileContentMap represents a mapping of file paths to file contents. Paths are relative to the \`holos\` output "deploy" directory, and may contain sub\-directories.
```go
typeFileContentMapmap[FilePath]FileContent
```
<a name="FilePath"></a>
## type FilePath {#FilePath}
FilePath represents a file path.
```go
typeFilePathstring
```
<a name="HelmChart"></a>
## type HelmChart {#HelmChart}
HelmChart represents a holos component which wraps around an upstream helm chart. Holos orchestrates helm by providing values obtained from CUE, renders the output using \`helm template\`, then post\-processes the helm output yaml using the general functionality provided by [HolosComponent](<#HolosComponent>), for example [Kustomize](<#Kustomize>) post\-rendering and mixing in additional kubernetes api objects.
```go
typeHelmChartstruct{
HolosComponent`json:",inline"`
Kindstring`json:"kind" cue:"\"HelmChart\""`
// Chart represents a helm chart to manage.
ChartChart`json:"chart"`
// ValuesContent represents the values.yaml file holos passes to the `helm
// template` command.
ValuesContentstring`json:"valuesContent"`
// EnableHooks enables helm hooks when executing the `helm template` command.
// Kustomize represents a kubectl kustomize build post-processing step.
Kustomize`json:"kustomize,omitempty"`
// Skip causes holos to take no action regarding this component.
Skipbool`json:"skip" cue:"bool | *false"`
}
```
<a name="Kind"></a>
## type Kind {#Kind}
Kind is a kubernetes api object kind. Defined as a type for clarity and type checking.
```go
typeKindstring
```
<a name="KubernetesObjects"></a>
## type KubernetesObjects {#KubernetesObjects}
KubernetesObjects represents a [HolosComponent](<#HolosComponent>) composed of Kubernetes API objects provided directly from CUE using [APIObjects](<#APIObjects>).
Kustomize represents resources necessary to execute a kustomize build. Intended for at least two use cases:
1. Process a [KustomizeBuild](<#KustomizeBuild>) [HolosComponent](<#HolosComponent>) which represents raw yaml file resources in a holos component directory.
2. Post process a [HelmChart](<#HelmChart>) [HolosComponent](<#HolosComponent>) to inject istio, patch jobs, add custom labels, etc...
```go
typeKustomizestruct{
// KustomizeFiles holds file contents for kustomize, e.g. patch files.
KustomizeBuild represents a [HolosComponent](<#HolosComponent>) that renders plain yaml files in the holos component directory using \`kubectl kustomize build\`.
```go
typeKustomizeBuildstruct{
HolosComponent`json:",inline"`
Kindstring`json:"kind" cue:"\"KustomizeBuild\""`
}
```
<a name="Label"></a>
## type Label {#Label}
Label is an arbitrary unique identifier internal to holos itself. The holos cli is expected to never write a Label value to rendered output files, therefore use a [Label](<#Label>) then the identifier must be unique and internal. Defined as a type for clarity and type checking.
A Label is useful to convert a CUE struct to a list, for example producing a list of [APIObject](<#APIObject>) resources from an [APIObjectMap](<#APIObjectMap>). A CUE struct using Label keys is guaranteed to not lose data when rendering output because a Label is expected to never be written to the final output.
```go
typeLabelstring
```
<a name="Metadata"></a>
## type Metadata {#Metadata}
Metadata represents data about the holos component such as the Name.
```go
typeMetadatastruct{
// Name represents the name of the holos component.
Namestring`json:"name"`
// Namespace is the primary namespace of the holos component. A holos
// component may manage resources in multiple namespaces, in this case
// consider setting the component namespace to default.
//
// This field is optional because not all resources require a namespace,
// particularly CRD's and DeployFiles functionality.
// +optional
Namespacestring`json:"namespace,omitempty"`
}
```
<a name="Platform"></a>
## type Platform {#Platform}
Platform represents a platform to manage. A Platform resource informs holos which components to build. The platform resource also acts as a container for the platform model form values provided by the PlatformService. The primary use case is to collect the cluster names, cluster types, platform model, and holos components to build into one resource.
```go
typePlatformstruct{
// Kind is a string value representing the resource this object represents.
Kindstring`json:"kind" cue:"\"Platform\""`
// APIVersion represents the versioned schema of this representation of an object.
// Metadata represents data about the object such as the Name.
MetadataPlatformMetadata`json:"metadata"`
// Spec represents the specification.
SpecPlatformSpec`json:"spec"`
}
```
<a name="PlatformMetadata"></a>
## type PlatformMetadata {#PlatformMetadata}
```go
typePlatformMetadatastruct{
// Name represents the Platform name.
Namestring`json:"name"`
}
```
<a name="PlatformSpec"></a>
## type PlatformSpec {#PlatformSpec}
PlatformSpec represents the specification of a Platform. Think of a platform specification as a list of platform components to apply to a list of kubernetes clusters combined with the user\-specified Platform Model.
```go
typePlatformSpecstruct{
// Model represents the platform model holos gets from from the
// PlatformService.GetPlatform rpc method and provides to CUE using a tag.
Modelstructpb.Struct`json:"model"`
// Components represents a list of holos components to manage.
Holos leverages a simple web app to collect and store platform attributes with a web form. Register an account with the web app to create and retrieve the platform model.
```
holos register user
```
:::tip
Holos allows you to customize all of the sections and fields of your platform model.
:::
## Generate your Platform
Generate your platform configuration from the holos reference platform embedded in the `holos` executable. Platform configuration is stored in a git repository.
```bash
mkdir holos-infra
cd holos-infra
holos generate platform holos
```
The generate command writes many files organized by platform component into the current directory
TODO: Put a table here describing key elements?
:::tip
Take a peek at `holos generate platform --help` to see other platforms embedded in the holos executable.
:::
## Push the Platform Form
```
holos push platform form .
```
## Fill in the form
TODO
## Pull the Platform Model
Once the platform model is saved, pull it into the holos-infra repository:
```
holos pull platform model .
```
## Render the Platform
With the platform model and the platform spec, you're ready to render the complete platform configuration:
- Provides a web form to pass top level parameters.
2. Reduces errors by eliminating the need to template unstructured text.
3. Is composable and scales down to a local machine.
4. Provides an way to safely configure broad authentication and authorization
policy.
## Next Steps
Dive deeper with the following resources that build on the foundation you have now.
1. Explore the [Rendering Process](/docs/design/rendering) in Holos.
2. Dive deeper into the [Platform Manifests](./platform-manifests) rendered in this guide.
3. Deploy [ArgoCD](/docs/guides/argocd) onto the foundation you built.
4. Deploy [Backstage](/docs/guides/backstage) as a portal to the integrated platform components.
## Clean-Up
If you'd like to clean up the resources you created in this guide, remove them
with:
```bash
k3d cluster delete workload
rm -rf holos-k3d
```
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
