Compare commits

...

18 Commits

Author SHA1 Message Date
Jeff McCune
16a6447926 helm: support oci images in chart name
Without this patch we do not support installing Kargo from an OCI helm
chart.  We want to support:

```
Component: #Helm & {
	Name:      "kargo"
	Namespace: Kargo.Namespace

	Chart: {
		name:    "oci://ghcr.io/akuity/kargo-charts/kargo"
		version: "1.0.3"
		release: Name
	}
	EnableHooks: true

	Values: Kargo.Values
}
```

This patch fixes the problem by using the base name for filesystem cache
operations.
2024-12-03 12:15:06 -08:00
Jeff McCune
111a5944ff cue: bump to 0.11.0
go get cuelang.org/go/cmd/cue@latest
2024-12-02 12:37:19 -08:00
Jeff McCune
ff1446dc93 docs: redirect /docs/guides/helm/
This shows up in the Unity tests I'm working on with mvdan and goes to a
blank page without the redirect in place.

	--- FAIL: TestGuides_v1alpha5 (0.00s)
	   --- FAIL: TestGuides_v1alpha5/helm (0.60s)
	       testscript.go:584: # Helm Guide https://holos.run/docs/guides/helm/
2024-12-02 09:05:13 -08:00
Jeff McCune
67ef990c37 v0.101.2 build tags 2024-12-02 08:09:23 -08:00
Jeff McCune
6bd54ab856 render: pass build tags from platform to component (#366)
Previously, build tags were not propagated from `holos render platform
-t validate` through to the underlying `holos render component` command.
This is a problem because validators need to be selectively enabled as a
work around until we have an audit mode field.

This patch fixes the problem by propagating command line tags from the
render platform command to the underlying commands.  This patch also
propagates tags for the show command.
2024-11-30 20:56:11 -08:00
Jeff McCune
89a23a10fd docs: remove DIRECTORY from holos render platform --help
The directory argument is deprecated now, use the --platform flag
instead.
2024-11-30 13:08:01 -08:00
Jeff McCune
5a939bb6fe render: support cue build tags e.g. -t foo for @if(foo) (#366)
Previously Holos only supported tags in the form of key=value.  CUE
supports boolean style tags in the form of `key [ "=" value ]` which we
want to use to conditionally use to register components with the
platform.

This patch modifies the flag parsing to support -t foo like cue does,
for use with the @if(foo) build tag.
2024-11-30 12:50:31 -08:00
Jeff McCune
ee16f14e03 refactor build plan pipeline
Previously the BuildPlan pipeline didn't execute generators and
transformers concurrently.  All steps were sequentially executed.  Holos
was primarily concurrent by executing multiple BuildPlans at once.

This patch changes the Build implementation for each BuildPlan to
execute a GoRoutine pipeline.  One producer fans out to a group of
routines each executing the pipeline for one artifact in the build plan.
The pipeline has 3 stages:

1: Fan-out to build each Generator concurrently.
2: Fan-in to build each Transformer sequentially.
3: Fan-out again to run each validator concurrently.

When the artifact pipelines return, the producer closes the tasks
channel causing the worker tasks to return.

Note the overall runtime for 8 BuildPlans is roughly equivalent to
previously at 160ms with --concurrency=8 on my M3 Max.  I expect this to
perform better than previously when multiple artifacts are rendered for
each BuildPlan.
2024-11-29 14:52:06 -08:00
Jeff McCune
7530345620 main: add tracing and profiling
Writes files based on parent pid and process pid to avoid collisions.

Analyze with:

export HOLOS_TRACE=trace.%d.%d.out
go tool trace trace.999.1000.out

export HOLOS_CPU_PROFILE=cpu.%d.%d.prof
go tool pprof cpu.999.1000.prof

export HOLOS_MEM_PROFILE=mem.%d.%d.prof
go tool pprof mem.999.1000.prof
2024-11-29 14:52:06 -08:00
Jeff McCune
47d60ef86d docs: fix cue vet path in validators post (#357)
Without this patch the validator fails if a component manages two of the
same kind of resource, which is common.

This patch updates the example to use the metadata namespace and name as
lookup keys.  This works for most components, but may not for
ClusterResources.  Use the kind top level field in that case and pass
the field name of the validator as a tag value to vary by component.
2024-11-25 19:35:02 -08:00
Jeff McCune
9e9f6efd04 docs: fix typos in validators blog post (#357) 2024-11-25 15:46:54 -08:00
Jeff McCune
fb4a043823 docs: add card for validators blog post (#357) 2024-11-25 15:11:11 -08:00
Jeff McCune
d718ab1910 docs: redirect /docs/local-cluster to the v1alpha5 topic 2024-11-25 10:53:31 -08:00
Jeff McCune
c649db18a9 docs: redirect /docs/quickstart to the overview 2024-11-25 10:43:16 -08:00
Jeff McCune
b3bddf3ee3 docs: add validators blog post (#357) 2024-11-25 08:49:27 -08:00
Jeff McCune
77836be250 docs: update readme 2024-11-25 08:10:11 -08:00
Jeff McCune
4db670b854 docs: add validators tutorial (#357)
Add a tutorial page on validators.
2024-11-24 21:34:09 -08:00
Jeff McCune
d87c919519 docs: redirect /docs/topics to structures
/docs/v1alpha5/api/author/ links to it in the opening paragraphs.
2024-11-24 19:38:30 -08:00
24 changed files with 1203 additions and 434 deletions

129
README.md
View File

@@ -1,35 +1,130 @@
## Holos - A Holistic Development Platform
# Holos
<img width="50%"
align="right"
style="display: block; margin: 40px auto;"
src="https://openinfrastructure.co/blog/2016/02/27/logo/logorectangle.png">
Building and maintaining a software development platform is a complex and time
consuming endeavour. Organizations often dedicate a team of 3-4 who need 6-12
months to build the platform.
[Holos] is a configuration management tool for Kubernetes implementing the
[rendered manifests pattern]. It handles configurations ranging from single
resources to multi-cluster platforms across regions.
Holos is a tool and a reference platform to reduce the complexity and speed up
the process of building a modern, cloud native software development platform.
Key components:
- Platform schemas defining component integration
- Building blocks unifying Helm, Kustomize and Kubernetes configs with CUE
- BuildPlan pipeline for generating, transforming and validating manifests
- **Accelerate new projects** - Reduce time to market and operational complexity by starting your new project on top of the Holos reference platform.
- **Modernize existing projects** - Incrementally incorporate your existing platform services into Holos for simpler integration.
- **Unified configuration model** - Increase safety and reduce the risk of config changes with CUE.
- **First class Helm and Kustomize support** - Leverage and reuse your existing investment in existing configuration tools such as Helm and Kustomize.
- **Modern Authentication and Authorization** - Holos seamlessly integrates platform identity and access management with zero-trust beyond corp style authorization policy.
```mermaid
---
title: Rendering Overview
---
graph LR
Platform[<a href="https://holos.run/docs/v1alpha5/api/author/#Platform">Platform</a>]
Component[<a href="https://holos.run/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
## Quick Installation
Helm[<a href="https://holos.run/docs/v1alpha5/api/author/#Helm">Helm</a>]
Kustomize[<a href="https://holos.run/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
Kubernetes[<a href="https://holos.run/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
```console
go install github.com/holos-run/holos/cmd/holos@latest
BuildPlan[<a href="https://holos.run/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
ResourcesArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
GitOpsArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
Generators[<a href="https://holos.run/docs/v1alpha5/api/core/#Generator">Generators</a>]
Transformers[<a href="https://holos.run/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
Validators[<a href="https://holos.run/docs/v1alpha5/api/core/#Validator">Validators</a>]
Files[Manifest<br/>Files]
Platform --> Component
Component --> Helm --> BuildPlan
Component --> Kubernetes --> BuildPlan
Component --> Kustomize --> BuildPlan
BuildPlan --> ResourcesArtifact --> Generators
BuildPlan --> GitOpsArtifact --> Generators
Generators --> Transformers --> Validators --> Files
```
## Docs and Support
## Setup
The documentation for developing and using Holos is available at: https://holos.run
```shell
brew install holos-run/tap/holos
```
For discussion and support, [open a discussion](https://github.com/orgs/holos-run/discussions/new/choose).
Refer to [setup] for other installation methods and dependencies.
## Example
See our [tutorial] for a complete hello world example.
```cue showLineNumbers
package holos
holos: Component.BuildPlan
Component: #Helm & {
Name: "podinfo"
Chart: {
version: "6.6.2"
repository: {
name: "podinfo"
url: "https://stefanprodan.github.io/podinfo"
}
}
Values: ui: {
message: string | *"Hello World" @tag(message, type=string)
}
}
```
## Organizational Role
Platform engineers use Holos to generate Kubernetes manifests, both locally and
in CI pipelines. The manifests are committed to version control and deployed via
GitOps tools like ArgoCD or Flux.
Holos integrates seamlessly with existing Helm charts, Kustomize bases, and
other version-controlled configurations.
## Advantages of Holos
### Safe
Holos leverages [CUE] for strong typing and validation of configuration data,
ensuring consistent output from Helm and other tools.
### Consistent
A unified pipeline processes all configurations - whether from CUE, Helm, or
Kustomize - through the same well-defined stages.
### Flexible
Composable building blocks for generation, transformation, validation and
integration let teams assemble workflows that match their needs.
The core is intentionally unopinionated about platform configuration patterns.
Common needs like environments and clusters are provided as customizable
[topics] recipes rather than enforced structures.
## Getting Help
Get support through our [Discord] channel or [GitHub discussions]. Configuration
challenges arise at all experience levels - we welcome your questions and are
here to help.
## License
Holos is licensed under Apache 2.0 as found in the [LICENSE file](LICENSE).
[Holos]: https://holos.run
[rendered manifests pattern]: https://akuity.io/blog/the-rendered-manifests-pattern
[CUE]: https://cuelang.org/
[Discord]: https://discord.gg/JgDVbNpye7
[GitHub discussions]: https://github.com/holos-run/holos/discussions
[Why CUE for Configuration]: https://holos.run/blog/why-cue-for-configuration/
[topics]: https://holos.run/docs/topics/
[setup]: https://holos.run/docs/setup/
[tutorial]: https://holos.run/docs/tutorial/

View File

@@ -209,7 +209,10 @@ type FilePath string
type FileContent string
// Validator validates files. Useful to validate an [Artifact] prior to writing
// it out to the final destination. Validators may be executed concurrently.
// it out to the final destination. Holos may execute validators concurrently.
// See the [validators] tutorial for an end to end example.
//
// [validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
type Validator struct {
// Kind represents the kind of transformer. Must be Kustomize, or Join.
Kind string `json:"kind" yaml:"kind" cue:"\"Command\""`

View File

@@ -7,11 +7,11 @@ cd $WORK
exec holos generate platform v1alpha5 --force
# Platforms are empty by default.
exec holos render platform ./platform
exec holos render platform
stderr -count=1 '^rendered platform'
# Holos uses CUE to build a platform specification.
exec cue export --expression holos --out=yaml ./platform
exec holos show platform
cmp stdout want/1.platform_spec.yaml
# Define the host and port in projects/blackbox.schema.cue
@@ -22,7 +22,7 @@ mv projects/platform/components/prometheus/prometheus.cue.disabled projects/plat
mv platform/prometheus.cue.disabled platform/prometheus.cue
# Render the platform to render the prometheus chart.
exec holos render platform ./platform
exec holos render platform
stderr -count=1 '^rendered prometheus'
stderr -count=1 '^rendered platform'
cmp deploy/components/prometheus/prometheus.gen.yaml want/1.prometheus.gen.yaml
@@ -73,8 +73,8 @@ core.#BuildPlan & {
metadata: name: _Tags.component.name
}
-- want/1.platform_spec.yaml --
kind: Platform
apiVersion: v1alpha5
kind: Platform
metadata:
name: default
spec:

View File

@@ -0,0 +1,50 @@
# https://github.com/holos-run/holos/issues/366
# Build tags conditionally include CUE files.
env HOME=$WORK
exec holos init platform v1alpha5 --force
exec holos show platform
cmp stdout want/empty.yaml
exec holos show platform -t foo
cmp stdout want/foo.yaml
-- platform/empty.cue --
@if(foo)
package holos
Platform: Components: foo: _
-- platform/metadata.cue --
package holos
Platform: Components: [NAME=string]: {
name: NAME
path: "components/empty"
labels: "app.holos.run/name": NAME
annotations: "app.holos.run/description": "\(NAME) empty test case"
}
-- components/empty/empty.cue --
package holos
Component: #Kubernetes & {}
holos: Component.BuildPlan
-- want/empty.yaml --
apiVersion: v1alpha5
kind: Platform
metadata:
name: default
spec:
components: []
-- want/foo.yaml --
apiVersion: v1alpha5
kind: Platform
metadata:
name: default
spec:
components:
- annotations:
app.holos.run/description: foo empty test case
labels:
app.holos.run/name: foo
name: foo
path: components/empty

View File

@@ -418,7 +418,7 @@ type Transformer struct {
<a name="Validator"></a>
## type Validator {#Validator}
Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Validators may be executed concurrently.
Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Holos may execute validators concurrently. See the [validators](<https://holos.run/docs/v1alpha5/tutorial/validators/>) tutorial for an end to end example.
```go
type Validator struct {

View File

@@ -265,9 +265,6 @@ This tutorial uses the `#Resources` structure to map resource kinds to their
schema definitions in CUE. This structure is defined in `resources.cue` at the
root of the tree. Take a look at [resources.cue] to see this mapping structure.
Continue to the next tutorial to learn how to define your own data structures
similar to the `#Resources` structure.
[Local Cluster]: ../topics/local-cluster.mdx
[ExternalSecret]: https://external-secrets.io/latest/api/externalsecret/
[Artifact]: ../api/core.md#Artifact

View File

@@ -11,7 +11,7 @@ import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
## Overview
Holos is a configuration management tool for Kubernetes that implements the
Holos is a configuration management tool for Kubernetes implementing the
[rendered manifests pattern]. It handles configurations ranging from single
resources to multi-cluster platforms across regions.

View File

@@ -0,0 +1,428 @@
---
slug: validators
title: Validators
description: Validate rendered manifests against policy definitions.
sidebar_position: 60
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
# Validators
## Overview
Sometimes Helm charts render Secrets we do not wanted committed to version
control for security. Helm charts often render incorrect manifests, even if
they're accepted by the api server. For example, passing `null` to collection
fields. We'll solve both of these issues using a [Validator] to block artifacts
with a Secret resource, and verifying the artifact against Kubernetes type
definitions.
1. If a Helm chart renders a Secret, Holos errors before writing the artifact
and suggests an ExternalSecret instead.
2. Each resource is validated against a field named by the value of the kind
field. For example, a `kind: Secret` resource validates against `secret: {}` in
CUE. `kind: Deployment` validates against `deployment: {}` in CUE.
3. The final artifact is validated, covering the output of all generators and
transformers.
<RenderingOverview />
## The Code
### Generating the Structure
Use `holos` to generate a minimal platform directory structure. First, create
and navigate into a blank directory. Then, use the `holos generate platform`
command to generate a minimal platform.
```shell
mkdir holos-validators-tutorial && cd holos-validators-tutorial
holos init platform v1alpha5
```
### Creating the Component
Create the directory for the `podinfo` component. Create an empty file, then add
the following CUE configuration to it.
```bash
mkdir -p components/podinfo
```
```bash
cat <<EOF > components/podinfo/podinfo.cue
```
```cue showLineNumbers
package holos
// export the component build plan to holos
holos: Component.BuildPlan
// Component is a Helm chart
Component: #Helm & {
Name: "podinfo"
Namespace: "default"
// Add metadata.namespace to all resources with kustomize.
KustomizeConfig: Kustomization: namespace: Namespace
Chart: {
version: "6.6.2"
repository: {
name: "podinfo"
url: "https://stefanprodan.github.io/podinfo"
}
}
}
```
```bash
EOF
```
Register the component with the platform.
```bash
cat <<EOF > platform/podinfo.cue
```
```cue showLineNumbers
package holos
Platform: Components: podinfo: {
name: "podinfo"
path: "components/podinfo"
}
```
```bash
EOF
```
Render the platform.
<Tabs groupId="tutorial-hello-render-manifests">
<TabItem value="command" label="Command">
```bash
holos render platform
```
</TabItem>
<TabItem value="output" label="Output">
```
cached podinfo 6.6.2
rendered podinfo in 1.938665041s
rendered platform in 1.938759417s
```
</TabItem>
</Tabs>
Add and commit the initial configuration.
```bash
git init . && git add . && git commit -m initial
```
### Define the Valid Schema
We'll use a CUE package named `policy` so the entire platform configuration in
package `holos` isn't loaded every time we validate an artifact.
Create `policy/validation-schema.cue` with the following content.
```shell
mkdir -p policy
cat <<EOF > policy/validation-schema.cue
```
```cue showLineNumbers
package policy
import apps "k8s.io/api/apps/v1"
// Organize by kind then name to avoid conflicts.
kind: [KIND=string]: [NAME=string]: {...}
// Useful when one component manages the same resource kind and name across
// multiple namespaces.
let KIND = kind
namespace: [NS=string]: KIND
// Block Secret resources. kind will not unify with "Secret"
kind: secret: [NAME=string]: kind: "Use an ExternalSecret instead. Forbidden by security policy. secret/\(NAME)"
// Validate Deployment against Kubernetes type definitions.
kind: deployment: [_]: apps.#Deployment
```
```shell
EOF
```
### Configuring Validators
Configure the Validators [ComponentConfig] field to configure each [BuildPlan]
to validate the rendered [Artifact] files.
```shell
cat <<EOF > validators.cue
```
```cue showLineNumbers
package holos
// Configure all component kinds to validate against the policy directory.
#ComponentConfig: Validators: cue: {
kind: "Command"
// Note --path maps each resource to a top level field named by the kind.
command: args: [
"holos",
"cue",
"vet",
"./policy",
"--path=\"namespace\"",
"--path=metadata.namespace",
"--path=strings.ToLower(kind)",
"--path=metadata.name",
]
}
```
```shell
EOF
```
### Patching Errors
Render the platform to see validation fail. The podinfo chart has no Secret,
but it produces an invalid Deployment because it sets the container resource
limits field to `null`.
```shell
holos render platform
```
```txt
deployment.spec.template.spec.containers.0.resources.limits: conflicting values null and {[string]:"k8s.io/apimachinery/pkg/api/resource".#Quantity} (mismatched types null and struct):
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:355:9
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:376:12
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2840:11
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2968:14
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:15
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:18
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:5027:9
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:6407:16
./policy/validation-schema.cue:9:13
../../../../../var/folders/22/T/holos.validate1636392304/components/podinfo/podinfo.gen.yaml:104:19
could not run: terminating because of errors
could not run: could not validate podinfo path ./components/podinfo: could not run command: holos cue vet ./policy --path strings.ToLower(kind) /var/folders/22/T/holos.validate1636392304/components/podinfo/podinfo.gen.yaml: exit status 1 at builder/v1alpha5/builder.go:411
could not run: could not render component: could not run command: holos --log-level info --log-format console render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo: exit status 1 at cli/render/render.go:155
```
We'll use a [Kustomize] patch [Transformer] to replace the `null` limits field
with a valid equivalent value.
:::important
This configuration is defined in CUE, not YAML, even though we're configuring a
Kustomize patch transformer. CUE gives us access to the unified platform
configuration.
:::
```shell
cat <<EOF > components/podinfo/patch.cue
```
```cue showLineNumbers
package holos
import "encoding/yaml"
Component: KustomizeConfig: Kustomization: {
_patches: limits: {
target: kind: "Deployment"
patch: yaml.Marshal([{
op: "test"
path: "/spec/template/spec/containers/0/resources/limits"
value: null
}, {
op: "replace"
path: "/spec/template/spec/containers/0/resources/limits"
value: {}
}])
}
patches: [for x in _patches {x}]
}
```
```shell
EOF
```
Now the platform renders.
<Tabs groupId="3A050092-8E56-49D4-84A9-71E544A21276">
<TabItem value="command" label="Command">
```bash
holos render platform
```
</TabItem>
<TabItem value="output" label="Output">
```txt
rendered podinfo in 181.875083ms
rendered platform in 181.975833ms
```
</TabItem>
</Tabs>
## Inspecting the BuildPlan
The BuildPlan patches the output of the upstream helm chart without modifying
it, then validates the artifact against the Kubernetes type definitions.
<Tabs groupId="1DAB4C46-0793-4CCA-8930-7B2E60BDA1BE">
<TabItem value="command" label="Command">
```bash
holos show buildplans
```
</TabItem>
<TabItem value="output" label="Output">
```yaml showLineNumbers
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: podinfo
spec:
artifacts:
- artifact: components/podinfo/podinfo.gen.yaml
generators:
- kind: Helm
output: helm.gen.yaml
helm:
chart:
name: podinfo
version: 6.6.2
release: podinfo
repository:
name: podinfo
url: https://stefanprodan.github.io/podinfo
values: {}
namespace: default
- kind: Resources
output: resources.gen.yaml
transformers:
- kind: Kustomize
inputs:
- helm.gen.yaml
- resources.gen.yaml
output: components/podinfo/podinfo.gen.yaml
kustomize:
kustomization:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
patches:
- patch: |
- op: test
path: /spec/template/spec/containers/0/resources/limits
value: null
- op: replace
path: /spec/template/spec/containers/0/resources/limits
value: {}
target:
kind: Deployment
name: ""
resources:
- helm.gen.yaml
- resources.gen.yaml
validators:
- kind: Command
inputs:
- components/podinfo/podinfo.gen.yaml
command:
args:
- holos
- cue
- vet
- ./policy
- --path
- strings.ToLower(kind)
```
</TabItem>
</Tabs>
## Catching Mistakes
Suppose a teammate downloads a helm chart that includes a Secret unbeknown to
them. Holos catches the problem and suggests an ExternalSecret instead.
Mix in a Secret to see what happens
```shell
cat <<EOF > components/podinfo/secret.cue
```
```cue showLineNumbers
package holos
Component: Resources: Secret: example: metadata: name: "example"
```
```shell
EOF
```
Render the platform to see the error.
```shell
holos render platform
```
```txt
secret.kind: conflicting values "Use an ExternalSecret instead. Forbidden by security policy." and "Secret":
./policy/validation-schema.cue:6:15
../../../../../var/folders/22/T/holos.validate2549739170/components/podinfo/podinfo.gen.yaml:1:7
could not run: terminating because of errors
could not run: could not validate podinfo path ./components/podinfo: could not run command: holos cue vet ./policy --path strings.ToLower(kind) /var/folders/22/T/holos.validate2549739170/components/podinfo/podinfo.gen.yaml: exit status 1 at builder/v1alpha5/builder.go:411
could not run: could not render component: could not run command: holos --log-level info --log-format console render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo: exit status 1 at cli/render/render.go:155
```
:::important
Holos quickly returns an error if validated artifacts have a Secret.
:::
Remove the secret to resolve the issue.
```shell
rm components/podinfo/secret.cue
```
## Inspecting the diff
The validation and patch results in a correct Deployment, verified against the
Kubernetes type definitions.
```shell
git diff
```
```diff
diff --git a/deploy/components/podinfo/podinfo.gen.yaml b/deploy/components/podinfo/podinfo.gen.yaml
index 6e4aec0..a145e3f 100644
--- a/deploy/components/podinfo/podinfo.gen.yaml
+++ b/deploy/components/podinfo/podinfo.gen.yaml
@@ -101,7 +101,7 @@ spec:
successThreshold: 1
timeoutSeconds: 5
resources:
- limits: null
+ limits: {}
requests:
cpu: 1m
memory: 16Mi
```
## Trying Locally
Optionally, apply the manifests rendered by Holos to a [Local Cluster] for
testing.
[Local Cluster]: ../topics/local-cluster.mdx
[ExternalSecret]: https://external-secrets.io/latest/api/externalsecret/
[Artifact]: ../api/core.md#Artifact
[BuildPlan]: ../api/core.md#BuildPlan
[Resources]: ../api/core.md#Resources
[Validator]: ../api/core.md#Validator
[Transformer]: ../api/core.md#Transformer
[Kustomize]: ../api/core.md#Kustomize
[Generator]: ../api/core.md#Generator
[Hello Holos]: ./hello-holos.mdx
[cue.mod/gen/external-secrets.io/externalsecret/v1beta1/types_gen.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/cue.mod/gen/external-secrets.io/externalsecret/v1beta1/types_gen.cue#L13
[ComponentConfig]: ../api/author.md#ComponentConfig
[timoni]: https://timoni.sh/install/
[resources.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/v1alpha5/resources.cue#L33

View File

@@ -0,0 +1,35 @@
---
slug: validators-feature
title: Validators added in Holos v0.101.0
authors: [jeff]
tags: [holos, feature]
image: /img/cards/validators.png
description: Validators are useful to enforce policy and catch Helm errors.
---
import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
import RenderPlatformDiagram from '@site/src/diagrams/render-platform-sequence.mdx';
We've added support for [Validators] in [v0.101.0]. Validators are useful to
enforce policies and ensure consistency early in the process. This feature
addresses two primary use cases:
1. Prevent insecure configuration early in the process. For example, prevent
Helm from rendering a `Secret` which would otherwise be committed to version control.
2. Prevent unsafe configuration by validating manifests against Kubernetes core
and custom resource type definitions.
Check out the [Validators] tutorial for examples of both use cases.
[Validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
[v0.101.0]: https://github.com/holos-run/holos/releases/tag/v0.101.0
{/* truncate */}
<RenderingOverview />
Validators complete the core functionality of the Holos manifest rendering
pipeline. We are seeking design partners to help enhance Generators and
Transformers. Validators are implemented using a generic `Command` kind, and
we're considering a similar kind for Generators and Transformers. Please
connect with us if you'd like to help design these enhancements.

View File

@@ -3,21 +3,21 @@
title: Rendering Overview
---
graph LR
Platform[<a href="/docs/v1alpha5/api/author/#Platform">Platform</a>]
Component[<a href="/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
Platform[<a href="https://holos.run/docs/v1alpha5/api/author/#Platform">Platform</a>]
Component[<a href="https://holos.run/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
Helm[<a href="/docs/v1alpha5/api/author/#Helm">Helm</a>]
Kustomize[<a href="/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
Kubernetes[<a href="/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
Helm[<a href="https://holos.run/docs/v1alpha5/api/author/#Helm">Helm</a>]
Kustomize[<a href="https://holos.run/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
Kubernetes[<a href="https://holos.run/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
BuildPlan[<a href="/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
BuildPlan[<a href="https://holos.run/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
ResourcesArtifact[<a href="/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
GitOpsArtifact[<a href="/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
ResourcesArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
GitOpsArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
Generators[<a href="/docs/v1alpha5/api/core/#Generator">Generators</a>]
Transformers[<a href="/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
Validators[Validators]
Generators[<a href="https://holos.run/docs/v1alpha5/api/core/#Generator">Generators</a>]
Transformers[<a href="https://holos.run/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
Validators[<a href="https://holos.run/docs/v1alpha5/api/core/#Validator">Validators</a>]
Files[Manifest<br/>Files]
Platform --> Component

View File

@@ -1,4 +1,16 @@
/docs /docs/v1alpha5/ 301
/docs/ /docs/v1alpha5/ 301
/docs/overview /docs/v1alpha5/tutorial/overview/ 301
/docs/overview/ /docs/v1alpha5/tutorial/overview/ 301
/docs /docs/v1alpha5/ 301
/docs/ /docs/v1alpha5/ 301
/docs/tutorial /docs/v1alpha5/tutorial/ 301
/docs/tutorial/ /docs/v1alpha5/tutorial/ 301
/docs/quickstart /docs/v1alpha5/tutorial/overview/ 301
/docs/quickstart/ /docs/v1alpha5/tutorial/overview/ 301
/docs/overview /docs/v1alpha5/tutorial/overview/ 301
/docs/overview/ /docs/v1alpha5/tutorial/overview/ 301
/docs/topics /docs/v1alpha5/topics/structures/ 301
/docs/topics/ /docs/v1alpha5/topics/structures/ 301
/docs/setup /docs/v1alpha5/tutorial/setup/ 301
/docs/setup/ /docs/v1alpha5/tutorial/setup/ 301
/docs/local-cluster /docs/v1alpha5/topics/local-cluster/ 301
/docs/local-cluster/ /docs/v1alpha5/topics/local-cluster/ 301
/docs/guides/helm /docs/v1alpha5/tutorial/helm-values/ 301
/docs/guides/helm/ /docs/v1alpha5/tutorial/helm-values/ 301

Binary file not shown.

After

Width:  |  Height:  |  Size: 98 KiB

24
go.mod
View File

@@ -10,7 +10,7 @@ require (
connectrpc.com/grpcreflect v1.2.0
connectrpc.com/otelconnect v0.7.0
connectrpc.com/validate v0.1.0
cuelang.org/go v0.10.1
cuelang.org/go v0.11.0
entgo.io/ent v0.13.1
github.com/bufbuild/buf v1.35.1
github.com/choria-io/machine-room v0.0.0-20240417064836-c604da2f005e
@@ -33,9 +33,9 @@ require (
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.9.0
golang.org/x/net v0.28.0
golang.org/x/net v0.30.0
golang.org/x/sync v0.8.0
golang.org/x/tools v0.24.0
golang.org/x/tools v0.26.0
google.golang.org/genproto/googleapis/rpc v0.0.0-20240617180043-68d350f18fd4
google.golang.org/protobuf v1.34.2
gopkg.in/yaml.v3 v3.0.1
@@ -53,7 +53,7 @@ require (
buf.build/gen/go/bufbuild/registry/connectrpc/go v1.16.2-20240610164129-660609bc46d3.1 // indirect
buf.build/gen/go/bufbuild/registry/protocolbuffers/go v1.34.2-20240610164129-660609bc46d3.2 // indirect
cloud.google.com/go/compute/metadata v0.3.0 // indirect
cuelabs.dev/go/oci/ociregistry v0.0.0-20240807094312-a32ad29eed79 // indirect
cuelabs.dev/go/oci/ociregistry v0.0.0-20240906074133-82eb438dd565 // indirect
dario.cat/mergo v1.0.0 // indirect
github.com/AlecAivazis/survey/v2 v2.3.7 // indirect
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible // indirect
@@ -256,7 +256,7 @@ require (
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/errors v0.9.1 // indirect
@@ -268,7 +268,7 @@ require (
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.52.3 // indirect
github.com/prometheus/procfs v0.13.0 // indirect
github.com/protocolbuffers/txtpbfmt v0.0.0-20231025115547-084445ff1adf // indirect
github.com/protocolbuffers/txtpbfmt v0.0.0-20240823084532-8e6b51fa9bef // indirect
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rivo/uniseg v0.4.7 // indirect
@@ -326,14 +326,14 @@ require (
go.uber.org/automaxprocs v1.5.3 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/crypto v0.26.0 // indirect
golang.org/x/crypto v0.28.0 // indirect
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8 // indirect
golang.org/x/exp/typeparams v0.0.0-20221208152030-732eee02a75a // indirect
golang.org/x/mod v0.20.0 // indirect
golang.org/x/oauth2 v0.22.0 // indirect
golang.org/x/sys v0.23.0 // indirect
golang.org/x/term v0.23.0 // indirect
golang.org/x/text v0.17.0 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/oauth2 v0.23.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/term v0.25.0 // indirect
golang.org/x/text v0.19.0 // indirect
golang.org/x/time v0.5.0 // indirect
google.golang.org/genproto v0.0.0-20240123012728-ef4313101c80 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240617180043-68d350f18fd4 // indirect

48
go.sum
View File

@@ -48,10 +48,10 @@ connectrpc.com/otelconnect v0.7.0 h1:ZH55ZZtcJOTKWWLy3qmL4Pam4RzRWBJFOqTPyAqCXkY
connectrpc.com/otelconnect v0.7.0/go.mod h1:Bt2ivBymHZHqxvo4HkJ0EwHuUzQN6k2l0oH+mp/8nwc=
connectrpc.com/validate v0.1.0 h1:r55jirxMK7HO/xZwVHj3w2XkVFarsUM77ZDy367NtH4=
connectrpc.com/validate v0.1.0/go.mod h1:GU47c9/x/gd+u9wRSPkrQOP46gx2rMN+Wo37EHgI3Ow=
cuelabs.dev/go/oci/ociregistry v0.0.0-20240807094312-a32ad29eed79 h1:EceZITBGET3qHneD5xowSTY/YHbNybvMWGh62K2fG/M=
cuelabs.dev/go/oci/ociregistry v0.0.0-20240807094312-a32ad29eed79/go.mod h1:5A4xfTzHTXfeVJBU6RAUf+QrlfTCW+017q/QiW+sMLg=
cuelang.org/go v0.10.1 h1:vDRRsd/5CICzisZ/13kBmXt3M+9eDl/pI06rrHyhlgA=
cuelang.org/go v0.10.1/go.mod h1:HzlaqqqInHNiqE6slTP6+UtxT9hN6DAzgJgdbNxXvX8=
cuelabs.dev/go/oci/ociregistry v0.0.0-20240906074133-82eb438dd565 h1:R5wwEcbEZSBmeyg91MJZTxfd7WpBo2jPof3AYjRbxwY=
cuelabs.dev/go/oci/ociregistry v0.0.0-20240906074133-82eb438dd565/go.mod h1:5A4xfTzHTXfeVJBU6RAUf+QrlfTCW+017q/QiW+sMLg=
cuelang.org/go v0.11.0 h1:2af2nhipqlUHtXk2dtOP5xnMm1ObGvKqIsJUJL1sRE4=
cuelang.org/go v0.11.0/go.mod h1:PBY6XvPUswPPJ2inpvUozP9mebDVTXaeehQikhZPBz0=
dario.cat/mergo v1.0.0 h1:AGCNq9Evsj31mOgNPcLyXc+4PNABt905YmuqPYYpBWk=
dario.cat/mergo v1.0.0/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
@@ -734,8 +734,8 @@ github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaR
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
github.com/pelletier/go-toml v1.9.5 h1:4yBQzkHv+7BHq2PQUZF3Mx0IYxG7LsP222s7Agd3ve8=
github.com/pelletier/go-toml v1.9.5/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M=
github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc=
github.com/pjbgf/sha1cd v0.3.0 h1:4D5XXmUUBUl/xQ6IjCkEAbqXskkq/4O7LmGn0AqMDs4=
github.com/pjbgf/sha1cd v0.3.0/go.mod h1:nZ1rrWOcGJ5uZgEEVL1VUM9iRQiZvWdbZjkKyFzPPsI=
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8/go.mod h1:HKlIX3XHQyzLZPlr7++PzdhaXEj94dEiJgZDTsxEqUI=
@@ -769,8 +769,8 @@ github.com/prometheus/common v0.52.3 h1:5f8uj6ZwHSscOGNdIQg6OiZv/ybiK2CO2q2drVZA
github.com/prometheus/common v0.52.3/go.mod h1:BrxBKv3FWBIGXw89Mg1AeBq7FSyRzXWI3l3e7W3RN5U=
github.com/prometheus/procfs v0.13.0 h1:GqzLlQyfsPbaEHaQkO7tbDlriv/4o5Hudv6OXHGKX7o=
github.com/prometheus/procfs v0.13.0/go.mod h1:cd4PFCR54QLnGKPaKGA6l+cfuNXtht43ZKY6tow0Y1g=
github.com/protocolbuffers/txtpbfmt v0.0.0-20231025115547-084445ff1adf h1:014O62zIzQwvoD7Ekj3ePDF5bv9Xxy0w6AZk0qYbjUk=
github.com/protocolbuffers/txtpbfmt v0.0.0-20231025115547-084445ff1adf/go.mod h1:jgxiZysxFPM+iWKwQwPR+y+Jvo54ARd4EisXxKYpB5c=
github.com/protocolbuffers/txtpbfmt v0.0.0-20240823084532-8e6b51fa9bef h1:ej+64jiny5VETZTqcc1GFVAPEtaSk6U1D0kKC2MS5Yc=
github.com/protocolbuffers/txtpbfmt v0.0.0-20240823084532-8e6b51fa9bef/go.mod h1:jgxiZysxFPM+iWKwQwPR+y+Jvo54ARd4EisXxKYpB5c=
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 h1:N/ElC8H3+5XpJzTSTfLsJV/mx9Q9g7kxmchpfZyxgzM=
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
@@ -968,8 +968,8 @@ golang.org/x/crypto v0.3.1-0.20221117191849-2c476679df9a/go.mod h1:hebNnKkNXi2Uz
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.26.0 h1:RrRspgV4mU+YwB4FYnuBoKsUapNIL5cohGAmSH3azsw=
golang.org/x/crypto v0.26.0/go.mod h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn54=
golang.org/x/crypto v0.28.0 h1:GBDwsMXVQi34v5CCYUm2jkJvu4cbtru2U4TN2PSyQnw=
golang.org/x/crypto v0.28.0/go.mod h1:rmgy+3RHxRZMyY0jjAJShp2zgEdOqj2AO7U0pYmeQ7U=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -1007,8 +1007,8 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.20.0 h1:utOm6MM3R3dnawAiJgn0y+xvuYRsm1RKM/4giyfDgV0=
golang.org/x/mod v0.20.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.21.0 h1:vvrHzRwRfVKSiLrG+d4FMl/Qi4ukBCE6kZlTUkDYRT0=
golang.org/x/mod v0.21.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -1048,16 +1048,16 @@ golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.28.0 h1:a9JDOJc5GMUJ0+UDqmLT86WiEy7iWyIhz8gz8E4e5hE=
golang.org/x/net v0.28.0/go.mod h1:yqtgsTWOOnlGLG9GFRrK3++bGOUEkNBoHZc8MEDWPNg=
golang.org/x/net v0.30.0 h1:AcW1SDZMkb8IpzCdQUaIq2sP4sZ4zw+55h6ynffypl4=
golang.org/x/net v0.30.0/go.mod h1:2wGyMJ5iFasEhkwi13ChkO/t1ECNC4X4eBKkVFyYFlU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20211005180243-6b3c2da341f1/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.22.0 h1:BzDx2FehcG7jJwgWLELCdmLuxk2i+x9UDpSiss2u0ZA=
golang.org/x/oauth2 v0.22.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/oauth2 v0.23.0 h1:PbgcYx2W7i4LvjJWEbf0ngHV6qJYr86PkAV3bXdLEbs=
golang.org/x/oauth2 v0.23.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -1138,8 +1138,8 @@ golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.23.0 h1:YfKFowiIMvtgl1UERQoTPPToxltDeZfbj4H7dVUCwmM=
golang.org/x/sys v0.23.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
@@ -1147,8 +1147,8 @@ golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.23.0 h1:F6D4vR+EHoL9/sWAWgAR1H2DcHr4PareCbAaCo1RpuU=
golang.org/x/term v0.23.0/go.mod h1:DgV24QBUrK6jhZXl+20l6UWznPlwAHm1Q1mGHtydmSk=
golang.org/x/term v0.25.0 h1:WtHI/ltw4NvSUig5KARz9h521QvRC8RmF/cuYqifU24=
golang.org/x/term v0.25.0/go.mod h1:RPyXicDX+6vLxogjjRxjgD2TKtmAO6NZBsBRfrOLu7M=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1163,8 +1163,8 @@ golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc=
golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
golang.org/x/text v0.19.0 h1:kTxAhCbGbxhK0IwgSKiMO5awPoDQ0RpfiVYBfK860YM=
golang.org/x/text v0.19.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -1217,8 +1217,8 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f
golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.24.0 h1:J1shsA93PJUEVaUSaay7UXAyE8aimq3GW0pjlolpa24=
golang.org/x/tools v0.24.0/go.mod h1:YhNqVBIfWHdzvTLs0d8LCuMhkKUgSUKldakyV7W/WDQ=
golang.org/x/tools v0.26.0 h1:v/60pFQmzmT9ExmjDv2gGIfi3OqfKoEP6I5+umXlbnQ=
golang.org/x/tools v0.26.0/go.mod h1:TPVVj70c7JJ3WCazhD8OdXcZg/og+b9+tH/KxylGwH0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

View File

@@ -20,6 +20,7 @@ var _ Store = NewStore()
type Store interface {
Get(path string) (data []byte, ok bool)
Set(path string, data []byte) error
// Save previously set path to dir preserving directories.
Save(dir, path string) error
}

View File

@@ -15,7 +15,7 @@ import (
// PlatformOpts represents build options when processing the components in a
// platform.
type PlatformOpts struct {
Fn BuildFunc
Fn func(context.Context, int, holos.Component) error
Selector holos.Selector
Concurrency int
InfoEnabled bool
@@ -89,9 +89,6 @@ func (p *Platform) Build(ctx context.Context, opts PlatformOpts) error {
return nil
}
// BuildFunc is executed concurrently when processing platform components.
type BuildFunc func(context.Context, int, holos.Component) error
func LoadPlatform(i *Instance) (platform Platform, err error) {
err = i.Discriminate(func(tm holos.TypeMeta) error {
if tm.Kind != "Platform" {

View File

@@ -5,16 +5,18 @@ import (
"context"
"encoding/json"
"fmt"
"io"
"log/slog"
"os"
"path"
"path/filepath"
"strings"
"sync"
"syscall"
"time"
"cuelang.org/go/cue"
core "github.com/holos-run/holos/api/core/v1alpha5"
"github.com/holos-run/holos/internal/artifact"
"github.com/holos-run/holos/internal/errors"
"github.com/holos-run/holos/internal/holos"
"github.com/holos-run/holos/internal/logger"
@@ -107,167 +109,84 @@ func (c *Component) Path() string {
}
var _ holos.BuildPlan = &BuildPlan{}
var _ task = generatorTask{}
var _ task = transformersTask{}
var _ task = validatorTask{}
// BuildPlan represents a component builder.
type BuildPlan struct {
core.BuildPlan
Opts holos.BuildOpts
type task interface {
id() string
run(ctx context.Context) error
}
// Build builds a BuildPlan into Artifact files.
func (b *BuildPlan) Build(ctx context.Context) error {
name := b.BuildPlan.Metadata.Name
path := b.Opts.Path
log := logger.FromContext(ctx).With("name", name, "path", path)
msg := fmt.Sprintf("could not build %s", name)
if b.BuildPlan.Spec.Disabled {
log.WarnContext(ctx, fmt.Sprintf("%s: disabled", msg))
return nil
}
type taskParams struct {
taskName string
buildPlanName string
opts holos.BuildOpts
}
g, ctx := errgroup.WithContext(ctx)
// One more for the producer
g.SetLimit(b.Opts.Concurrency + 1)
func (t taskParams) id() string {
return fmt.Sprintf("%s:%s/%s", t.opts.Path, t.buildPlanName, t.taskName)
}
// Producer.
g.Go(func() error {
for _, a := range b.BuildPlan.Spec.Artifacts {
msg := fmt.Sprintf("%s artifact %s", msg, a.Artifact)
log := log.With("artifact", a.Artifact)
if a.Skip {
log.WarnContext(ctx, fmt.Sprintf("%s: skipped field is true", msg))
continue
}
select {
case <-ctx.Done():
return ctx.Err()
default:
// https://golang.org/doc/faq#closures_and_goroutines
a := a
// Worker. Blocks if limit has been reached.
g.Go(func() error {
for _, gen := range a.Generators {
switch gen.Kind {
case "Resources":
if err := b.resources(log, gen, b.Opts.Store); err != nil {
return errors.Format("could not generate resources: %w", err)
}
case "Helm":
if err := b.helm(ctx, log, gen, b.Opts.Store); err != nil {
return errors.Format("could not generate helm: %w", err)
}
case "File":
if err := b.file(log, gen, b.Opts.Store); err != nil {
return errors.Format("could not generate file: %w", err)
}
default:
return errors.Format("%s: unsupported kind %s", msg, gen.Kind)
}
}
type generatorTask struct {
taskParams
generator core.Generator
wg *sync.WaitGroup
}
for _, t := range a.Transformers {
switch t.Kind {
case "Kustomize":
if err := b.kustomize(ctx, log, t, b.Opts.Store); err != nil {
return errors.Wrap(err)
}
case "Join":
s := make([][]byte, 0, len(t.Inputs))
for _, input := range t.Inputs {
if data, ok := b.Opts.Store.Get(string(input)); ok {
s = append(s, data)
} else {
return errors.Format("%s: missing %s", msg, input)
}
}
data := bytes.Join(s, []byte(t.Join.Separator))
if err := b.Opts.Store.Set(string(t.Output), data); err != nil {
return errors.Format("%s: %w", msg, err)
}
log.Debug("set artifact: " + string(t.Output))
default:
return errors.Format("%s: unsupported kind %s", msg, t.Kind)
}
}
for _, validator := range a.Validators {
switch validator.Kind {
case "Command":
if err := b.validate(ctx, log, validator, b.Opts.Store); err != nil {
return errors.Wrap(err)
}
default:
return errors.Format("%s: unsupported kind %s", msg, validator.Kind)
}
}
// Write the final artifact
if err := b.Opts.Store.Save(b.Opts.WriteTo, string(a.Artifact)); err != nil {
return errors.Format("%s: %w", msg, err)
}
log.DebugContext(ctx, "wrote "+filepath.Join(b.Opts.WriteTo, string(a.Artifact)))
return nil
})
}
func (t generatorTask) run(ctx context.Context) error {
defer t.wg.Done()
msg := fmt.Sprintf("could not build %s", t.id())
switch t.generator.Kind {
case "Resources":
if err := t.resources(); err != nil {
return errors.Format("%s: could not generate resources: %w", msg, err)
}
return nil
})
// Wait for completion and return the first error (if any)
return g.Wait()
}
func (b *BuildPlan) Export(idx int, encoder holos.OrderedEncoder) error {
if err := encoder.Encode(idx, &b.BuildPlan); err != nil {
return errors.Wrap(err)
case "Helm":
if err := t.helm(ctx); err != nil {
return errors.Format("%s: could not generate helm: %w", msg, err)
}
case "File":
if err := t.file(); err != nil {
return errors.Format("%s: could not generate file: %w", msg, err)
}
default:
return errors.Format("%s: unsupported kind %s", msg, t.generator.Kind)
}
return nil
}
func (b *BuildPlan) Load(v cue.Value) error {
return errors.Wrap(v.Decode(&b.BuildPlan))
}
func (b *BuildPlan) file(
log *slog.Logger,
g core.Generator,
store artifact.Store,
) error {
data, err := os.ReadFile(filepath.Join(string(b.Opts.Path), string(g.File.Source)))
func (t generatorTask) file() error {
data, err := os.ReadFile(filepath.Join(string(t.opts.Path), string(t.generator.File.Source)))
if err != nil {
return errors.Wrap(err)
}
if err := store.Set(string(g.Output), data); err != nil {
if err := t.opts.Store.Set(string(t.generator.Output), data); err != nil {
return errors.Wrap(err)
}
log.Debug("set artifact: " + string(g.Output))
return nil
}
func (b *BuildPlan) helm(
ctx context.Context,
log *slog.Logger,
g core.Generator,
store artifact.Store,
) error {
chartName := g.Helm.Chart.Name
log = log.With("chart", chartName)
func (t generatorTask) helm(ctx context.Context) error {
chartName := t.generator.Helm.Chart.Name
// Unnecessary? cargo cult copied from internal/cli/render/render.go
if chartName == "" {
return errors.New("missing chart name")
}
// Cache the chart by version to pull new versions. (#273)
cacheDir := filepath.Join(string(b.Opts.Path), "vendor", g.Helm.Chart.Version)
cacheDir := filepath.Join(string(t.opts.Path), "vendor", t.generator.Helm.Chart.Version)
cachePath := filepath.Join(cacheDir, filepath.Base(chartName))
log := logger.FromContext(ctx)
if _, err := os.Stat(cachePath); os.IsNotExist(err) {
timeout, cancel := context.WithTimeout(ctx, 5*time.Minute)
defer cancel()
err := onceWithLock(log, timeout, cachePath, func() error {
return b.cacheChart(ctx, log, cacheDir, g.Helm.Chart)
})
err := func() error {
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
defer cancel()
return onceWithLock(log, ctx, cachePath, func() error {
return cacheChart(ctx, cacheDir, t.generator.Helm.Chart, t.opts.Stderr)
})
}()
if err != nil {
return errors.Format("could not cache chart: %w", err)
}
@@ -280,7 +199,7 @@ func (b *BuildPlan) helm(
}
defer util.Remove(ctx, tempDir)
data, err := yaml.Marshal(g.Helm.Values)
data, err := yaml.Marshal(t.generator.Helm.Values)
if err != nil {
return errors.Format("could not marshal values: %w", err)
}
@@ -293,22 +212,22 @@ func (b *BuildPlan) helm(
// Run charts
args := []string{"template"}
if !g.Helm.EnableHooks {
if !t.generator.Helm.EnableHooks {
args = append(args, "--no-hooks")
}
for _, apiVersion := range g.Helm.APIVersions {
for _, apiVersion := range t.generator.Helm.APIVersions {
args = append(args, "--api-versions", apiVersion)
}
if kubeVersion := g.Helm.KubeVersion; kubeVersion != "" {
if kubeVersion := t.generator.Helm.KubeVersion; kubeVersion != "" {
args = append(args, "--kube-version", kubeVersion)
}
args = append(args,
"--include-crds",
"--values", valuesPath,
"--namespace", g.Helm.Namespace,
"--namespace", t.generator.Helm.Namespace,
"--kubeconfig", "/dev/null",
"--version", g.Helm.Chart.Version,
g.Helm.Chart.Release,
"--version", t.generator.Helm.Chart.Version,
t.generator.Helm.Chart.Release,
cachePath,
)
helmOut, err := util.RunCmd(ctx, "helm", args...)
@@ -325,36 +244,31 @@ func (b *BuildPlan) helm(
}
// Set the artifact
if err := store.Set(string(g.Output), helmOut.Stdout.Bytes()); err != nil {
if err := t.opts.Store.Set(string(t.generator.Output), helmOut.Stdout.Bytes()); err != nil {
return errors.Format("could not store helm output: %w", err)
}
log.Debug("set artifact: " + string(g.Output))
log.Debug("set artifact: " + string(t.generator.Output))
return nil
}
func (b *BuildPlan) resources(
log *slog.Logger,
g core.Generator,
store artifact.Store,
) error {
func (t generatorTask) resources() error {
var size int
for _, m := range g.Resources {
for _, m := range t.generator.Resources {
size += len(m)
}
list := make([]core.Resource, 0, size)
for _, m := range g.Resources {
for _, m := range t.generator.Resources {
for _, r := range m {
list = append(list, r)
}
}
msg := fmt.Sprintf(
"could not generate %s for %s path %s",
g.Output,
b.BuildPlan.Metadata.Name,
b.Opts.Path,
"could not generate %s for %s",
t.generator.Output,
t.id(),
)
buf, err := marshal(list)
@@ -362,114 +276,220 @@ func (b *BuildPlan) resources(
return errors.Format("%s: %w", msg, err)
}
if err := store.Set(string(g.Output), buf.Bytes()); err != nil {
return errors.Format("%s: %w", msg, err)
}
log.Debug("set artifact " + string(g.Output))
return nil
}
func (b *BuildPlan) validate(
ctx context.Context,
log *slog.Logger,
validator core.Validator,
store artifact.Store,
) error {
tempDir, err := os.MkdirTemp("", "holos.validate")
if err != nil {
return errors.Wrap(err)
}
defer util.Remove(ctx, tempDir)
msg := fmt.Sprintf(
"could not validate %s path %s",
b.BuildPlan.Metadata.Name,
b.Opts.Path,
)
// Write the inputs
for _, input := range validator.Inputs {
path := string(input)
if err := store.Save(tempDir, path); err != nil {
return errors.Format("%s: %w", msg, err)
}
log.DebugContext(ctx, "wrote "+filepath.Join(tempDir, path))
}
if len(validator.Command.Args) < 1 {
return errors.Format("%s: command args length must be at least 1", msg)
}
size := len(validator.Command.Args) + len(validator.Inputs)
args := make([]string, 0, size)
args = append(args, validator.Command.Args...)
for _, input := range validator.Inputs {
args = append(args, filepath.Join(tempDir, string(input)))
}
// Execute the validator
if _, err = util.RunCmdA(ctx, b.Opts.Stderr, args[0], args[1:]...); err != nil {
if err := t.opts.Store.Set(string(t.generator.Output), buf.Bytes()); err != nil {
return errors.Format("%s: %w", msg, err)
}
return nil
}
func (b *BuildPlan) kustomize(
ctx context.Context,
log *slog.Logger,
t core.Transformer,
store artifact.Store,
) error {
tempDir, err := os.MkdirTemp("", "holos.kustomize")
if err != nil {
return errors.Wrap(err)
}
defer util.Remove(ctx, tempDir)
msg := fmt.Sprintf(
"could not transform %s for %s path %s",
t.Output,
b.BuildPlan.Metadata.Name,
b.Opts.Path,
)
type transformersTask struct {
taskParams
transformers []core.Transformer
wg *sync.WaitGroup
}
// Write the kustomization
data, err := yaml.Marshal(t.Kustomize.Kustomization)
if err != nil {
return errors.Format("%s: %w", msg, err)
}
path := filepath.Join(tempDir, "kustomization.yaml")
if err := os.WriteFile(path, data, 0666); err != nil {
return errors.Format("%s: %w", msg, err)
}
log.DebugContext(ctx, "wrote "+path)
// Write the inputs
for _, input := range t.Inputs {
path := string(input)
if err := store.Save(tempDir, path); err != nil {
return errors.Format("%s: %w", msg, err)
func (t transformersTask) run(ctx context.Context) error {
defer t.wg.Done()
for idx, transformer := range t.transformers {
msg := fmt.Sprintf("could not build %s/%d", t.id(), idx)
switch transformer.Kind {
case "Kustomize":
if err := kustomize(ctx, transformer, t.taskParams); err != nil {
return errors.Wrap(err)
}
case "Join":
s := make([][]byte, 0, len(transformer.Inputs))
for _, input := range transformer.Inputs {
if data, ok := t.opts.Store.Get(string(input)); ok {
s = append(s, data)
} else {
return errors.Format("%s: missing %s", msg, input)
}
}
data := bytes.Join(s, []byte(transformer.Join.Separator))
if err := t.opts.Store.Set(string(transformer.Output), data); err != nil {
return errors.Format("%s: %w", msg, err)
}
default:
return errors.Format("%s: unsupported kind %s", msg, transformer.Kind)
}
log.DebugContext(ctx, "wrote "+filepath.Join(tempDir, path))
}
return nil
}
// Execute kustomize
r, err := util.RunCmdW(ctx, b.Opts.Stderr, "kubectl", "kustomize", tempDir)
if err != nil {
kErr := r.Stderr.String()
err = errors.Format("%s: could not run kustomize: %w", msg, err)
log.ErrorContext(ctx, fmt.Sprintf("%s: stderr:\n%s", err.Error(), kErr), "err", err, "stderr", kErr)
return err
type validatorTask struct {
taskParams
validator core.Validator
wg *sync.WaitGroup
}
func (t validatorTask) run(ctx context.Context) error {
defer t.wg.Done()
msg := fmt.Sprintf("could not validate %s", t.id())
switch kind := t.validator.Kind; kind {
case "Command":
if err := validate(ctx, t.validator, t.taskParams); err != nil {
return errors.Wrap(err)
}
default:
return errors.Format("%s: unsupported kind %s", msg, kind)
}
return nil
}
// Store the artifact
if err := store.Set(string(t.Output), r.Stdout.Bytes()); err != nil {
func worker(ctx context.Context, idx int, tasks chan task) error {
log := logger.FromContext(ctx).With("worker", idx)
for {
select {
case <-ctx.Done():
return ctx.Err()
case task, ok := <-tasks:
if !ok {
log.DebugContext(ctx, fmt.Sprintf("worker %d returning: tasks chan closed", idx))
return nil
}
log.DebugContext(ctx, fmt.Sprintf("worker %d task %s starting", idx, task.id()))
if err := task.run(ctx); err != nil {
return errors.Wrap(err)
}
log.DebugContext(ctx, fmt.Sprintf("worker %d task %s finished ok", idx, task.id()))
}
}
}
func buildArtifact(ctx context.Context, idx int, artifact core.Artifact, tasks chan task, buildPlanName string, opts holos.BuildOpts) error {
var wg sync.WaitGroup
msg := fmt.Sprintf("could not build %s artifact %s", buildPlanName, artifact.Artifact)
// Process Generators concurrently
for gid, gen := range artifact.Generators {
task := generatorTask{
taskParams: taskParams{
taskName: fmt.Sprintf("artifact/%d/generator/%d", idx, gid),
buildPlanName: buildPlanName,
opts: opts,
},
generator: gen,
wg: &wg,
}
wg.Add(1)
select {
case <-ctx.Done():
return ctx.Err()
case tasks <- task:
}
}
wg.Wait()
// Process Transformers sequentially
task := transformersTask{
taskParams: taskParams{
taskName: fmt.Sprintf("artifact/%d/transformers", idx),
buildPlanName: buildPlanName,
opts: opts,
},
transformers: artifact.Transformers,
wg: &wg,
}
wg.Add(1)
select {
case <-ctx.Done():
return ctx.Err()
case tasks <- task:
}
wg.Wait()
// Process Validators concurrently
for vid, val := range artifact.Validators {
task := validatorTask{
taskParams: taskParams{
taskName: fmt.Sprintf("artifact/%d/validator/%d", idx, vid),
buildPlanName: buildPlanName,
opts: opts,
},
validator: val,
wg: &wg,
}
wg.Add(1)
select {
case <-ctx.Done():
return ctx.Err()
case tasks <- task:
}
}
wg.Wait()
// Write the final artifact
out := string(artifact.Artifact)
if err := opts.Store.Save(opts.WriteTo, out); err != nil {
return errors.Format("%s: %w", msg, err)
}
log.Debug("set artifact " + string(t.Output))
log := logger.FromContext(ctx)
log.DebugContext(ctx, fmt.Sprintf("wrote %s", filepath.Join(opts.WriteTo, out)))
return nil
}
// BuildPlan represents a component builder.
type BuildPlan struct {
core.BuildPlan
Opts holos.BuildOpts
}
func (b *BuildPlan) Build(ctx context.Context) error {
name := b.BuildPlan.Metadata.Name
path := b.Opts.Path
log := logger.FromContext(ctx).With("name", name, "path", path)
msg := fmt.Sprintf("could not build %s", name)
if b.BuildPlan.Spec.Disabled {
log.WarnContext(ctx, fmt.Sprintf("%s: disabled", msg))
return nil
}
g, ctx := errgroup.WithContext(ctx)
tasks := make(chan task)
// Start the worker pool.
for idx := 0; idx < max(1, b.Opts.Concurrency); idx++ {
g.Go(func() error {
return worker(ctx, idx, tasks)
})
}
// Start one producer that fans out to one pipeline per artifact.
g.Go(func() error {
// Close the tasks chan when the producer returns.
defer func() {
log.DebugContext(ctx, "producer returning: closing tasks chan")
close(tasks)
}()
// Separate error group for producers.
p, ctx := errgroup.WithContext(ctx)
for idx, a := range b.BuildPlan.Spec.Artifacts {
p.Go(func() error {
return buildArtifact(ctx, idx, a, tasks, b.Metadata.Name, b.Opts)
})
}
// Wait on producers to finish.
return errors.Wrap(p.Wait())
})
// Wait on workers to finish.
return g.Wait()
}
func (b *BuildPlan) Export(idx int, encoder holos.OrderedEncoder) error {
if err := encoder.Encode(idx, &b.BuildPlan); err != nil {
return errors.Wrap(err)
}
return nil
}
func (b *BuildPlan) Load(v cue.Value) error {
return errors.Wrap(v.Decode(&b.BuildPlan))
}
func marshal(list []core.Resource) (buf bytes.Buffer, err error) {
encoder := yaml.NewEncoder(&buf)
defer encoder.Close()
@@ -482,95 +502,6 @@ func marshal(list []core.Resource) (buf bytes.Buffer, err error) {
return
}
// cacheChart stores a cached copy of Chart in the chart subdirectory of path.
//
// We assume the only method responsible for writing to chartDir is cacheChart
// itself. cacheChart runs concurrently when rendering a platform.
//
// We rely on the atomicity of moving temporary directories into place on the
// same filesystem via os.Rename. If a syscall.EEXIST error occurs during
// renaming, it indicates that the cached chart already exists, which is
// expected when this function is called concurrently.
//
// TODO(jeff): Break the dependency on v1alpha5, make it work across versions as
// a utility function.
func (b *BuildPlan) cacheChart(
ctx context.Context,
log *slog.Logger,
cacheDir string,
chart core.Chart,
) error {
// Add repositories
repo := chart.Repository
stderr := b.Opts.Stderr
if repo.URL == "" {
// repo update not needed for oci charts so this is debug instead of warn.
log.DebugContext(ctx, "skipped helm repo add and update: repo url is empty")
} else {
if _, err := util.RunCmdW(ctx, stderr, "helm", "repo", "add", repo.Name, repo.URL); err != nil {
return errors.Format("could not run helm repo add: %w", err)
}
if _, err := util.RunCmdW(ctx, stderr, "helm", "repo", "update", repo.Name); err != nil {
return errors.Format("could not run helm repo update: %w", err)
}
}
cacheTemp, err := os.MkdirTemp(cacheDir, chart.Name)
if err != nil {
return errors.Wrap(err)
}
defer util.Remove(ctx, cacheTemp)
cn := chart.Name
if chart.Repository.Name != "" {
cn = fmt.Sprintf("%s/%s", chart.Repository.Name, chart.Name)
}
helmOut, err := util.RunCmdW(ctx, stderr, "helm", "pull", "--destination", cacheTemp, "--untar=true", "--version", chart.Version, cn)
if err != nil {
stderr := helmOut.Stderr.String()
lines := strings.Split(stderr, "\n")
for _, line := range lines {
log.DebugContext(ctx, line)
if strings.HasPrefix(line, "Error:") {
err = fmt.Errorf("%s: %w", line, err)
}
}
return errors.Format("could not run helm pull: %w", err)
}
log.Debug("helm pull", "stdout", helmOut.Stdout, "stderr", helmOut.Stderr)
items, err := os.ReadDir(cacheTemp)
if err != nil {
return errors.Wrap(fmt.Errorf("could not read directory: %w", err))
}
if len(items) != 1 {
return errors.Format("want: exactly one item, have: %+v", items)
}
item := items[0]
src := filepath.Join(cacheTemp, item.Name())
dst := filepath.Join(cacheDir, chart.Name)
if err := os.Rename(src, dst); err != nil {
var linkErr *os.LinkError
if errors.As(err, &linkErr) && errors.Is(linkErr.Err, syscall.EEXIST) {
log.DebugContext(ctx, "cache already exists", "chart", chart.Name, "chart_version", chart.Version, "path", dst)
} else {
return errors.Wrap(fmt.Errorf("could not rename: %w", err))
}
} else {
log.DebugContext(ctx, fmt.Sprintf("renamed %s to %s", src, dst), "src", src, "dst", dst)
}
log.InfoContext(ctx,
fmt.Sprintf("cached %s %s", chart.Name, chart.Version),
"chart", chart.Name,
"chart_version", chart.Version,
"path", dst,
)
return nil
}
// onceWithLock obtains a filesystem lock with mkdir, then executes fn. If the
// lock is already locked, onceWithLock waits for it to be released then returns
// without calling fn.
@@ -619,3 +550,158 @@ func onceWithLock(log *slog.Logger, ctx context.Context, path string, fn func()
// Unexpected error
return errors.Wrap(err)
}
func cacheChart(ctx context.Context, cacheDir string, chart core.Chart, stderr io.Writer) error {
log := logger.FromContext(ctx)
// Add repositories
repo := chart.Repository
if repo.URL == "" {
// repo update not needed for oci charts so this is debug instead of warn.
log.DebugContext(ctx, "skipped helm repo add and update: repo url is empty")
} else {
if _, err := util.RunCmdW(ctx, stderr, "helm", "repo", "add", repo.Name, repo.URL); err != nil {
return errors.Format("could not run helm repo add: %w", err)
}
if _, err := util.RunCmdW(ctx, stderr, "helm", "repo", "update", repo.Name); err != nil {
return errors.Format("could not run helm repo update: %w", err)
}
}
// Support chart.Name = "oci:/ghcr.io/akuity/kargo-charts/kargo"
chartBaseName := path.Base(chart.Name)
cacheTemp, err := os.MkdirTemp(cacheDir, chartBaseName)
if err != nil {
return errors.Wrap(err)
}
defer util.Remove(ctx, cacheTemp)
cn := chart.Name
if chart.Repository.Name != "" {
cn = fmt.Sprintf("%s/%s", chart.Repository.Name, chart.Name)
}
helmOut, err := util.RunCmdW(ctx, stderr, "helm", "pull", "--destination", cacheTemp, "--untar=true", "--version", chart.Version, cn)
if err != nil {
stderr := helmOut.Stderr.String()
lines := strings.Split(stderr, "\n")
for _, line := range lines {
log.DebugContext(ctx, line)
if strings.HasPrefix(line, "Error:") {
err = fmt.Errorf("%s: %w", line, err)
}
}
return errors.Format("could not run helm pull: %w", err)
}
log.Debug("helm pull", "stdout", helmOut.Stdout, "stderr", helmOut.Stderr)
items, err := os.ReadDir(cacheTemp)
if err != nil {
return errors.Wrap(fmt.Errorf("could not read directory: %w", err))
}
if len(items) != 1 {
return errors.Format("want: exactly one item, have: %+v", items)
}
item := items[0]
src := filepath.Join(cacheTemp, item.Name())
dst := filepath.Join(cacheDir, chartBaseName)
if err := os.Rename(src, dst); err != nil {
var linkErr *os.LinkError
if errors.As(err, &linkErr) && errors.Is(linkErr.Err, syscall.EEXIST) {
log.DebugContext(ctx, "cache already exists", "chart", chart.Name, "chart_version", chart.Version, "path", dst)
} else {
return errors.Wrap(fmt.Errorf("could not rename: %w", err))
}
} else {
log.DebugContext(ctx, fmt.Sprintf("renamed %s to %s", src, dst), "src", src, "dst", dst)
}
log.InfoContext(ctx,
fmt.Sprintf("cached %s %s", chart.Name, chart.Version),
"chart", chart.Name,
"chart_version", chart.Version,
"path", dst,
)
return nil
}
func kustomize(ctx context.Context, t core.Transformer, p taskParams) error {
tempDir, err := os.MkdirTemp("", "holos.kustomize")
if err != nil {
return errors.Wrap(err)
}
defer util.Remove(ctx, tempDir)
msg := fmt.Sprintf(
"could not transform %s for %s path %s",
t.Output,
p.buildPlanName,
p.opts.Path,
)
// Write the kustomization
data, err := yaml.Marshal(t.Kustomize.Kustomization)
if err != nil {
return errors.Format("%s: %w", msg, err)
}
path := filepath.Join(tempDir, "kustomization.yaml")
if err := os.WriteFile(path, data, 0666); err != nil {
return errors.Format("%s: %w", msg, err)
}
// Write the inputs
for _, input := range t.Inputs {
path := string(input)
if err := p.opts.Store.Save(tempDir, path); err != nil {
return errors.Format("%s: %w", msg, err)
}
}
// Execute kustomize
r, err := util.RunCmdW(ctx, p.opts.Stderr, "kubectl", "kustomize", tempDir)
if err != nil {
return errors.Format("%s: could not run kustomize: %w", msg, err)
}
// Store the artifact
if err := p.opts.Store.Set(string(t.Output), r.Stdout.Bytes()); err != nil {
return errors.Format("%s: %w", msg, err)
}
return nil
}
func validate(ctx context.Context, validator core.Validator, p taskParams) error {
store := p.opts.Store
tempDir, err := os.MkdirTemp("", "holos.validate")
if err != nil {
return errors.Wrap(err)
}
// defer util.Remove(ctx, tempDir)
msg := fmt.Sprintf("could not validate %s", p.id())
// Write the inputs
for _, input := range validator.Inputs {
path := string(input)
if err := store.Save(tempDir, path); err != nil {
return errors.Format("%s: %w", msg, err)
}
}
if len(validator.Command.Args) < 1 {
return errors.Format("%s: command args length must be at least 1", msg)
}
size := len(validator.Command.Args) + len(validator.Inputs)
args := make([]string, 0, size)
args = append(args, validator.Command.Args...)
for _, input := range validator.Inputs {
args = append(args, filepath.Join(tempDir, string(input)))
}
// Execute the validator
if _, err = util.RunCmdA(ctx, p.opts.Stderr, args[0], args[1:]...); err != nil {
return errors.Format("%s: %w", msg, err)
}
return nil
}

View File

@@ -4,6 +4,9 @@ import (
"context"
"fmt"
"log/slog"
"os"
"runtime/pprof"
"runtime/trace"
"connectrpc.com/connect"
cue "cuelang.org/go/cue/errors"
@@ -12,12 +15,48 @@ import (
"google.golang.org/genproto/googleapis/rpc/errdetails"
)
func memProfile(ctx context.Context, cfg *holos.Config) {
if format := os.Getenv("HOLOS_MEM_PROFILE"); format != "" {
f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
defer f.Close()
if err := pprof.WriteHeapProfile(f); err != nil {
_ = HandleError(ctx, err, cfg)
}
}
}
// MakeMain makes a main function for the cli or tests.
func MakeMain(options ...holos.Option) func() int {
return func() (exitCode int) {
cfg := holos.New(options...)
slog.SetDefault(cfg.Logger())
ctx := context.Background()
if format := os.Getenv("HOLOS_CPU_PROFILE"); format != "" {
f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
err := pprof.StartCPUProfile(f)
defer func() {
pprof.StopCPUProfile()
f.Close()
}()
if err != nil {
return HandleError(ctx, err, cfg)
}
}
defer memProfile(ctx, cfg)
if format := os.Getenv("HOLOS_TRACE"); format != "" {
f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
err := trace.Start(f)
defer func() {
trace.Stop()
f.Close()
}()
if err != nil {
return HandleError(ctx, err, cfg)
}
}
feature := &holos.EnvFlagger{}
if err := New(cfg, feature).ExecuteContext(ctx); err != nil {
return HandleError(ctx, err, cfg)

View File

@@ -16,6 +16,8 @@ import (
"github.com/spf13/cobra"
)
const tagHelp = "set the value of a cue @tag field in the form key [ = value ]"
func New(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
cmd := command.New("render")
cmd.Args = cobra.NoArgs
@@ -26,7 +28,7 @@ func New(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
}
func newPlatform(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
cmd := command.New("platform DIRECTORY")
cmd := command.New("platform")
cmd.Args = cobra.MaximumNArgs(1)
cmd.Example = "holos render platform"
cmd.Short = "render an entire platform"
@@ -38,13 +40,13 @@ func newPlatform(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
}
var concurrency int
cmd.Flags().IntVar(&concurrency, "concurrency", min(runtime.NumCPU(), 8), "number of components to render concurrently")
cmd.Flags().IntVar(&concurrency, "concurrency", runtime.NumCPU(), "number of components to render concurrently")
var platform string
cmd.Flags().StringVar(&platform, "platform", "./platform", "platform directory path")
var selector holos.Selector
cmd.Flags().VarP(&selector, "selector", "l", "label selector (e.g. label==string,label!=string)")
tagMap := make(holos.TagMap)
cmd.Flags().VarP(&tagMap, "inject", "t", "set the value of a cue @tag field from a key=value pair")
cmd.Flags().VarP(&tagMap, "inject", "t", tagHelp)
cmd.RunE = func(cmd *cobra.Command, args []string) error {
ctx := cmd.Root().Context()
@@ -70,7 +72,7 @@ func newPlatform(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
"--log-format", cfg.LogConfig().Format(),
}
opts := builder.PlatformOpts{
Fn: makePlatformRenderFunc(cmd.ErrOrStderr(), prefixArgs),
Fn: makeComponentRenderFunc(cmd.ErrOrStderr(), prefixArgs, tagMap.Tags()),
Selector: selector,
Concurrency: concurrency,
InfoEnabled: true,
@@ -102,9 +104,9 @@ func newComponent(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
}
tagMap := make(holos.TagMap)
cmd.Flags().VarP(&tagMap, "inject", "t", "set the value of a cue @tag field from a key=value pair")
cmd.Flags().VarP(&tagMap, "inject", "t", tagHelp)
var concurrency int
cmd.Flags().IntVar(&concurrency, "concurrency", min(runtime.NumCPU(), 8), "number of concurrent build steps")
cmd.Flags().IntVar(&concurrency, "concurrency", runtime.NumCPU(), "number of concurrent build steps")
cmd.RunE = func(cmd *cobra.Command, args []string) error {
ctx := cmd.Root().Context()
@@ -134,7 +136,7 @@ func newComponent(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
return cmd
}
func makePlatformRenderFunc(w io.Writer, prefixArgs []string) builder.BuildFunc {
func makeComponentRenderFunc(w io.Writer, prefixArgs, cliTags []string) func(context.Context, int, holos.Component) error {
return func(ctx context.Context, idx int, component holos.Component) error {
select {
case <-ctx.Done():
@@ -147,6 +149,9 @@ func makePlatformRenderFunc(w io.Writer, prefixArgs []string) builder.BuildFunc
args := make([]string, 0, 10+len(prefixArgs)+(len(tags)*2))
args = append(args, prefixArgs...)
args = append(args, "render", "component")
for _, tag := range cliTags {
args = append(args, "--inject", tag)
}
for _, tag := range tags {
args = append(args, "--inject", tag)
}

View File

@@ -94,6 +94,7 @@ func newShowBuildPlanCmd() (cmd *cobra.Command) {
buildPlanOpts := holos.NewBuildOpts(path)
buildPlanOpts.Stderr = cmd.ErrOrStderr()
buildPlanOpts.Concurrency = concurrency
buildPlanOpts.Tags = tagMap.Tags()
platformOpts := builder.PlatformOpts{
Fn: makeBuildFunc(encoder, buildPlanOpts),
@@ -110,7 +111,7 @@ func newShowBuildPlanCmd() (cmd *cobra.Command) {
return cmd
}
func makeBuildFunc(encoder holos.OrderedEncoder, opts holos.BuildOpts) builder.BuildFunc {
func makeBuildFunc(encoder holos.OrderedEncoder, opts holos.BuildOpts) func(context.Context, int, holos.Component) error {
return func(ctx context.Context, idx int, component holos.Component) error {
select {
case <-ctx.Done():
@@ -120,6 +121,7 @@ func makeBuildFunc(encoder holos.OrderedEncoder, opts holos.BuildOpts) builder.B
if err != nil {
return errors.Wrap(err)
}
tags = append(tags, opts.Tags...)
inst, err := builder.LoadInstance(component.Path(), tags)
if err != nil {
return errors.Wrap(err)

View File

@@ -232,7 +232,10 @@ package core
#FileContent: string
// Validator validates files. Useful to validate an [Artifact] prior to writing
// it out to the final destination. Validators may be executed concurrently.
// it out to the final destination. Holos may execute validators concurrently.
// See the [validators] tutorial for an end to end example.
//
// [validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
#Validator: {
// Kind represents the kind of transformer. Must be Kustomize, or Join.
kind: string & "Command" @go(Kind)

View File

@@ -40,13 +40,22 @@ func (i *StringSlice) Set(value string) error {
return nil
}
// TagMap represents a map of key values for CUE TagMap for flag parsing.
type TagMap map[string]string
// TagMap represents a map of key values for CUE TagMap for flag parsing. The
// values are pointers to disambiguate between the case where a tag is a boolean
// ("--inject foo") and the case where a tag has a string zero value ("--inject
// foo="). Refer to the Tags field of [cue/load.Config]
//
// [cue/load.Config]: https://pkg.go.dev/cuelang.org/go@v0.10.1/cue/load#Config
type TagMap map[string]*string
func (t TagMap) Tags() []string {
parts := make([]string, 0, len(t))
for k, v := range t {
parts = append(parts, fmt.Sprintf("%s=%s", k, v))
for tag, val := range t {
if val == nil {
parts = append(parts, tag)
} else {
parts = append(parts, fmt.Sprintf("%s=%s", tag, *val))
}
}
return parts
}
@@ -60,10 +69,14 @@ func (t TagMap) String() string {
// is not supported.
func (t TagMap) Set(value string) error {
parts := strings.SplitN(value, "=", 2)
if len(parts) != 2 {
switch len(parts) {
case 1:
t[parts[0]] = nil
case 2:
t[parts[0]] = &parts[1]
default:
return errors.Format("invalid format, must be tag=value")
}
t[parts[0]] = parts[1]
return nil
}
@@ -307,6 +320,7 @@ type BuildOpts struct {
Stderr io.Writer
WriteTo string
Path string
Tags []string
}
func NewBuildOpts(path string) BuildOpts {
@@ -316,5 +330,6 @@ func NewBuildOpts(path string) BuildOpts {
Stderr: os.Stderr,
WriteTo: "deploy",
Path: path,
Tags: make([]string, 0, 10),
}
}

View File

@@ -43,10 +43,11 @@ func RunCmd(ctx context.Context, name string, args ...string) (result RunResult,
cmd.Stdout = result.Stdout
cmd.Stderr = result.Stderr
log := logger.FromContext(ctx)
log.DebugContext(ctx, "running: "+name, "name", name, "args", args)
command := fmt.Sprintf("%s '%s'", name, strings.Join(args, "' '"))
log.DebugContext(ctx, "running command: "+command, "name", name, "args", args)
err = cmd.Run()
if err != nil {
err = fmt.Errorf("could not run command: %s %s: %w", name, strings.Join(args, " "), err)
err = fmt.Errorf("could not run command:\n\t%s\n\t%w", command, err)
}
return result, err
}

View File

@@ -1 +1 @@
0
3