refactor build plan pipeline

Previously the BuildPlan pipeline didn't execute generators and
transformers concurrently.  All steps were sequentially executed.  Holos
was primarily concurrent by executing multiple BuildPlans at once.

This patch changes the Build implementation for each BuildPlan to
execute a GoRoutine pipeline.  One producer fans out to a group of
routines each executing the pipeline for one artifact in the build plan.
The pipeline has 3 stages:

1: Fan-out to build each Generator concurrently.
2: Fan-in to build each Transformer sequentially.
3: Fan-out again to run each validator concurrently.

When the artifact pipelines return, the producer closes the tasks
channel causing the worker tasks to return.

Note the overall runtime for 8 BuildPlans is roughly equivalent to
previously at 160ms with --concurrency=8 on my M3 Max.  I expect this to
perform better than previously when multiple artifacts are rendered for
each BuildPlan.

README.md

@@ -1,35 +1,130 @@
-## Holos - A Holistic Development Platform
+# Holos
 
 <img width="50%"
 align="right"
 style="display: block; margin: 40px auto;"
 src="https://openinfrastructure.co/blog/2016/02/27/logo/logorectangle.png">
 
-Building and maintaining a software development platform is a complex and time
-consuming endeavour.  Organizations often dedicate a team of 3-4 who need 6-12
-months to build the platform.
+[Holos] is a configuration management tool for Kubernetes implementing the
+[rendered manifests pattern]. It handles configurations ranging from single
+resources to multi-cluster platforms across regions.
 
-Holos is a tool and a reference platform to reduce the complexity and speed up
-the process of building a modern, cloud native software development platform.
+Key components:
+- Platform schemas defining component integration
+- Building blocks unifying Helm, Kustomize and Kubernetes configs with CUE
+- BuildPlan pipeline for generating, transforming and validating manifests
 
-- **Accelerate new projects** - Reduce time to market and operational complexity by starting your new project on top of the Holos reference platform.
-- **Modernize existing projects** - Incrementally incorporate your existing platform services into Holos for simpler integration.
-- **Unified configuration model** - Increase safety and reduce the risk of config changes with CUE.
-- **First class Helm and Kustomize support** - Leverage and reuse your existing investment in existing configuration tools such as Helm and Kustomize.
-- **Modern Authentication and Authorization** - Holos seamlessly integrates platform identity and access management with zero-trust beyond corp style authorization policy.
+```mermaid
+---
+title: Rendering Overview
+---
+graph LR
+    Platform[<a href="https://holos.run/docs/v1alpha5/api/author/#Platform">Platform</a>]
+    Component[<a href="https://holos.run/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
 
-## Quick Installation
+    Helm[<a href="https://holos.run/docs/v1alpha5/api/author/#Helm">Helm</a>]
+    Kustomize[<a href="https://holos.run/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
+    Kubernetes[<a href="https://holos.run/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
 
-```console
-go install github.com/holos-run/holos/cmd/holos@latest
+    BuildPlan[<a href="https://holos.run/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
+
+    ResourcesArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
+    GitOpsArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
+
+    Generators[<a href="https://holos.run/docs/v1alpha5/api/core/#Generator">Generators</a>]
+    Transformers[<a href="https://holos.run/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
+    Validators[<a href="https://holos.run/docs/v1alpha5/api/core/#Validator">Validators</a>]
+    Files[Manifest<br/>Files]
+
+    Platform --> Component
+    Component --> Helm --> BuildPlan
+    Component --> Kubernetes --> BuildPlan
+    Component --> Kustomize --> BuildPlan
+
+    BuildPlan --> ResourcesArtifact --> Generators
+    BuildPlan --> GitOpsArtifact --> Generators
+
+    Generators --> Transformers --> Validators --> Files
 ```
 
-## Docs and Support
+## Setup
 
-The documentation for developing and using Holos is available at: https://holos.run
+```shell
+brew install holos-run/tap/holos
+```
 
-For discussion and support, [open a discussion](https://github.com/orgs/holos-run/discussions/new/choose).
+Refer to [setup] for other installation methods and dependencies.
+
+## Example
+
+See our [tutorial] for a complete hello world example.
+
+```cue showLineNumbers
+package holos
+
+holos: Component.BuildPlan
+
+Component: #Helm & {
+	Name: "podinfo"
+	Chart: {
+		version: "6.6.2"
+		repository: {
+			name: "podinfo"
+			url:  "https://stefanprodan.github.io/podinfo"
+		}
+	}
+	Values: ui: {
+		message: string | *"Hello World" @tag(message, type=string)
+	}
+}
+```
+
+## Organizational Role
+
+Platform engineers use Holos to generate Kubernetes manifests, both locally and
+in CI pipelines. The manifests are committed to version control and deployed via
+GitOps tools like ArgoCD or Flux.
+
+Holos integrates seamlessly with existing Helm charts, Kustomize bases, and
+other version-controlled configurations.
+
+## Advantages of Holos
+
+### Safe
+
+Holos leverages [CUE] for strong typing and validation of configuration data,
+ensuring consistent output from Helm and other tools.
+
+### Consistent
+
+A unified pipeline processes all configurations - whether from CUE, Helm, or
+Kustomize - through the same well-defined stages.
+
+### Flexible
+
+Composable building blocks for generation, transformation, validation and
+integration let teams assemble workflows that match their needs.
+
+The core is intentionally unopinionated about platform configuration patterns.
+Common needs like environments and clusters are provided as customizable
+[topics] recipes rather than enforced structures.
+
+## Getting Help
+
+Get support through our [Discord] channel or [GitHub discussions]. Configuration
+challenges arise at all experience levels - we welcome your questions and are
+here to help.
 
 ## License
 
 Holos is licensed under Apache 2.0 as found in the [LICENSE file](LICENSE).
+
+[Holos]: https://holos.run
+[rendered manifests pattern]: https://akuity.io/blog/the-rendered-manifests-pattern
+[CUE]: https://cuelang.org/
+[Discord]: https://discord.gg/JgDVbNpye7
+[GitHub discussions]: https://github.com/holos-run/holos/discussions
+[Why CUE for Configuration]: https://holos.run/blog/why-cue-for-configuration/
+[topics]: https://holos.run/docs/topics/
+[setup]: https://holos.run/docs/setup/
+[tutorial]: https://holos.run/docs/tutorial/

api/core/v1alpha5/types.go

@@ -209,7 +209,10 @@ type FilePath string
 type FileContent string
 
 // Validator validates files.  Useful to validate an [Artifact] prior to writing
-// it out to the final destination.  Validators may be executed concurrently.
+// it out to the final destination.  Holos may execute validators concurrently.
+// See the [validators] tutorial for an end to end example.
+//
+// [validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
 type Validator struct {
 	// Kind represents the kind of transformer. Must be Kustomize, or Join.
 	Kind string `json:"kind" yaml:"kind" cue:"\"Command\""`

cmd/holos/tests/v1alpha5/guides/helm.txt

@@ -7,11 +7,11 @@ cd $WORK
 exec holos generate platform v1alpha5 --force
 
 # Platforms are empty by default.
-exec holos render platform ./platform
+exec holos render platform
 stderr -count=1 '^rendered platform'
 
 # Holos uses CUE to build a platform specification.
-exec cue export --expression holos --out=yaml ./platform
+exec holos show platform
 cmp stdout want/1.platform_spec.yaml
 
 # Define the host and port in projects/blackbox.schema.cue
@@ -22,7 +22,7 @@ mv projects/platform/components/prometheus/prometheus.cue.disabled projects/plat
 mv platform/prometheus.cue.disabled platform/prometheus.cue
 
 # Render the platform to render the prometheus chart.
-exec holos render platform ./platform
+exec holos render platform
 stderr -count=1 '^rendered prometheus'
 stderr -count=1 '^rendered platform'
 cmp deploy/components/prometheus/prometheus.gen.yaml want/1.prometheus.gen.yaml
@@ -73,8 +73,8 @@ core.#BuildPlan & {
 	metadata: name: _Tags.component.name
 }
 -- want/1.platform_spec.yaml --
-kind: Platform
 apiVersion: v1alpha5
+kind: Platform
 metadata:
   name: default
 spec:

doc/md/api/core.md

@@ -418,7 +418,7 @@ type Transformer struct {
 <a name="Validator"></a>
 ## type Validator {#Validator}
 
-Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Validators may be executed concurrently.
+Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Holos may execute validators concurrently. See the [validators](<https://holos.run/docs/v1alpha5/tutorial/validators/>) tutorial for an end to end example.
 
 ```go
 type Validator struct {

doc/md/tutorial/cue.mdx

@@ -265,9 +265,6 @@ This tutorial uses the `#Resources` structure to map resource kinds to their
 schema definitions in CUE. This structure is defined in `resources.cue` at the
 root of the tree. Take a look at [resources.cue] to see this mapping structure.
 
-Continue to the next tutorial to learn how to define your own data structures
-similar to the `#Resources` structure.
-
 [Local Cluster]: ../topics/local-cluster.mdx
 [ExternalSecret]: https://external-secrets.io/latest/api/externalsecret/
 [Artifact]: ../api/core.md#Artifact

doc/md/tutorial/overview.mdx

@@ -11,7 +11,7 @@ import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
 
 ## Overview
 
-Holos is a configuration management tool for Kubernetes that implements the
+Holos is a configuration management tool for Kubernetes implementing the
 [rendered manifests pattern]. It handles configurations ranging from single
 resources to multi-cluster platforms across regions.
 

doc/md/tutorial/validators.mdx

@@ -0,0 +1,428 @@
+---
+slug: validators
+title: Validators
+description: Validate rendered manifests against policy definitions.
+sidebar_position: 60
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
+
+# Validators
+
+## Overview
+
+Sometimes Helm charts render Secrets we do not wanted committed to version
+control for security.  Helm charts often render incorrect manifests, even if
+they're accepted by the api server.  For example, passing `null` to collection
+fields.  We'll solve both of these issues using a [Validator] to block artifacts
+with a Secret resource, and verifying the artifact against Kubernetes type
+definitions.
+
+1. If a Helm chart renders a Secret, Holos errors before writing the artifact
+and suggests an ExternalSecret instead.
+2. Each resource is validated against a field named by the value of the kind
+field.  For example, a `kind: Secret` resource validates against `secret: {}` in
+CUE.  `kind: Deployment` validates against `deployment: {}` in CUE.
+3. The final artifact is validated, covering the output of all generators and
+transformers.
+
+<RenderingOverview />
+
+## The Code
+
+### Generating the Structure
+
+Use `holos` to generate a minimal platform directory structure. First, create
+and navigate into a blank directory. Then, use the `holos generate platform`
+command to generate a minimal platform.
+
+```shell
+mkdir holos-validators-tutorial && cd holos-validators-tutorial
+holos init platform v1alpha5
+```
+
+### Creating the Component
+
+Create the directory for the `podinfo` component. Create an empty file, then add
+the following CUE configuration to it.
+
+```bash
+mkdir -p components/podinfo
+```
+```bash
+cat <<EOF > components/podinfo/podinfo.cue
+```
+```cue showLineNumbers
+package holos
+
+// export the component build plan to holos
+holos: Component.BuildPlan
+
+// Component is a Helm chart
+Component: #Helm & {
+	Name:      "podinfo"
+	Namespace: "default"
+	// Add metadata.namespace to all resources with kustomize.
+	KustomizeConfig: Kustomization: namespace: Namespace
+	Chart: {
+		version: "6.6.2"
+		repository: {
+			name: "podinfo"
+			url:  "https://stefanprodan.github.io/podinfo"
+		}
+	}
+}
+```
+```bash
+EOF
+```
+
+Register the component with the platform.
+
+```bash
+cat <<EOF > platform/podinfo.cue
+```
+```cue showLineNumbers
+package holos
+
+Platform: Components: podinfo: {
+	name: "podinfo"
+	path: "components/podinfo"
+}
+```
+```bash
+EOF
+```
+
+Render the platform.
+
+<Tabs groupId="tutorial-hello-render-manifests">
+  <TabItem value="command" label="Command">
+```bash
+holos render platform
+```
+  </TabItem>
+  <TabItem value="output" label="Output">
+```
+cached podinfo 6.6.2
+rendered podinfo in 1.938665041s
+rendered platform in 1.938759417s
+```
+  </TabItem>
+</Tabs>
+
+Add and commit the initial configuration.
+
+```bash
+git init . && git add . && git commit -m initial
+```
+
+### Define the Valid Schema
+
+We'll use a CUE package named `policy` so the entire platform configuration in
+package `holos` isn't loaded every time we validate an artifact.
+
+Create `policy/validation-schema.cue` with the following content.
+
+```shell
+mkdir -p policy
+cat <<EOF > policy/validation-schema.cue
+```
+```cue showLineNumbers
+package policy
+
+import apps "k8s.io/api/apps/v1"
+
+// Organize by kind then name to avoid conflicts.
+kind: [KIND=string]: [NAME=string]: {...}
+
+// Useful when one component manages the same resource kind and name across
+// multiple namespaces.
+let KIND = kind
+namespace: [NS=string]: KIND
+
+// Block Secret resources. kind will not unify with "Secret"
+kind: secret: [NAME=string]: kind: "Use an ExternalSecret instead.  Forbidden by security policy.  secret/\(NAME)"
+
+// Validate Deployment against Kubernetes type definitions.
+kind: deployment: [_]: apps.#Deployment
+```
+```shell
+EOF
+```
+
+### Configuring Validators
+
+Configure the Validators [ComponentConfig] field to configure each [BuildPlan]
+to validate the rendered [Artifact] files.
+
+```shell
+cat <<EOF > validators.cue
+```
+```cue showLineNumbers
+package holos
+
+// Configure all component kinds to validate against the policy directory.
+#ComponentConfig: Validators: cue: {
+	kind: "Command"
+	// Note --path maps each resource to a top level field named by the kind.
+	command: args: [
+		"holos",
+		"cue",
+		"vet",
+		"./policy",
+		"--path=\"namespace\"",
+		"--path=metadata.namespace",
+		"--path=strings.ToLower(kind)",
+		"--path=metadata.name",
+	]
+}
+```
+```shell
+EOF
+```
+
+### Patching Errors
+
+Render the platform to see validation fail.  The podinfo chart has no Secret,
+but it produces an invalid Deployment because it sets the container resource
+limits field to `null`.
+
+```shell
+holos render platform
+```
+
+```txt
+deployment.spec.template.spec.containers.0.resources.limits: conflicting values null and {[string]:"k8s.io/apimachinery/pkg/api/resource".#Quantity} (mismatched types null and struct):
+    ./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:355:9
+    ./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:376:12
+    ./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2840:11
+    ./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2968:14
+    ./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:15
+    ./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:18
+    ./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:5027:9
+    ./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:6407:16
+    ./policy/validation-schema.cue:9:13
+    ../../../../../var/folders/22/T/holos.validate1636392304/components/podinfo/podinfo.gen.yaml:104:19
+could not run: terminating because of errors
+could not run: could not validate podinfo path ./components/podinfo: could not run command: holos cue vet ./policy --path strings.ToLower(kind) /var/folders/22/T/holos.validate1636392304/components/podinfo/podinfo.gen.yaml: exit status 1 at builder/v1alpha5/builder.go:411
+could not run: could not render component: could not run command: holos --log-level info --log-format console render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo: exit status 1 at cli/render/render.go:155
+```
+
+We'll use a [Kustomize] patch [Transformer] to replace the `null` limits field
+with a valid equivalent value.
+
+:::important
+This configuration is defined in CUE, not YAML, even though we're configuring a
+Kustomize patch transformer.  CUE gives us access to the unified platform
+configuration.
+:::
+
+```shell
+cat <<EOF > components/podinfo/patch.cue
+```
+```cue showLineNumbers
+package holos
+
+import "encoding/yaml"
+
+Component: KustomizeConfig: Kustomization: {
+	_patches: limits: {
+		target: kind: "Deployment"
+		patch: yaml.Marshal([{
+			op:    "test"
+			path:  "/spec/template/spec/containers/0/resources/limits"
+			value: null
+		}, {
+			op:   "replace"
+			path: "/spec/template/spec/containers/0/resources/limits"
+			value: {}
+		}])
+	}
+	patches: [for x in _patches {x}]
+}
+```
+```shell
+EOF
+```
+
+Now the platform renders.
+
+<Tabs groupId="3A050092-8E56-49D4-84A9-71E544A21276">
+  <TabItem value="command" label="Command">
+```bash
+holos render platform
+```
+  </TabItem>
+  <TabItem value="output" label="Output">
+```txt
+rendered podinfo in 181.875083ms
+rendered platform in 181.975833ms
+```
+  </TabItem>
+</Tabs>
+
+## Inspecting the BuildPlan
+
+The BuildPlan patches the output of the upstream helm chart without modifying
+it, then validates the artifact against the Kubernetes type definitions.
+
+<Tabs groupId="1DAB4C46-0793-4CCA-8930-7B2E60BDA1BE">
+  <TabItem value="command" label="Command">
+```bash
+holos show buildplans
+```
+  </TabItem>
+  <TabItem value="output" label="Output">
+```yaml showLineNumbers
+kind: BuildPlan
+apiVersion: v1alpha5
+metadata:
+  name: podinfo
+spec:
+  artifacts:
+    - artifact: components/podinfo/podinfo.gen.yaml
+      generators:
+        - kind: Helm
+          output: helm.gen.yaml
+          helm:
+            chart:
+              name: podinfo
+              version: 6.6.2
+              release: podinfo
+              repository:
+                name: podinfo
+                url: https://stefanprodan.github.io/podinfo
+            values: {}
+            namespace: default
+        - kind: Resources
+          output: resources.gen.yaml
+      transformers:
+        - kind: Kustomize
+          inputs:
+            - helm.gen.yaml
+            - resources.gen.yaml
+          output: components/podinfo/podinfo.gen.yaml
+          kustomize:
+            kustomization:
+              apiVersion: kustomize.config.k8s.io/v1beta1
+              kind: Kustomization
+              namespace: default
+              patches:
+                - patch: |
+                    - op: test
+                      path: /spec/template/spec/containers/0/resources/limits
+                      value: null
+                    - op: replace
+                      path: /spec/template/spec/containers/0/resources/limits
+                      value: {}
+                  target:
+                    kind: Deployment
+                    name: ""
+              resources:
+                - helm.gen.yaml
+                - resources.gen.yaml
+      validators:
+        - kind: Command
+          inputs:
+            - components/podinfo/podinfo.gen.yaml
+          command:
+            args:
+              - holos
+              - cue
+              - vet
+              - ./policy
+              - --path
+              - strings.ToLower(kind)
+```
+  </TabItem>
+</Tabs>
+
+## Catching Mistakes
+
+Suppose a teammate downloads a helm chart that includes a Secret unbeknown to
+them.  Holos catches the problem and suggests an ExternalSecret instead.
+
+Mix in a Secret to see what happens
+
+```shell
+cat <<EOF > components/podinfo/secret.cue
+```
+```cue showLineNumbers
+package holos
+
+Component: Resources: Secret: example: metadata: name: "example"
+```
+```shell
+EOF
+```
+
+Render the platform to see the error.
+
+```shell
+holos render platform
+```
+```txt
+secret.kind: conflicting values "Use an ExternalSecret instead.  Forbidden by security policy." and "Secret":
+    ./policy/validation-schema.cue:6:15
+    ../../../../../var/folders/22/T/holos.validate2549739170/components/podinfo/podinfo.gen.yaml:1:7
+could not run: terminating because of errors
+could not run: could not validate podinfo path ./components/podinfo: could not run command: holos cue vet ./policy --path strings.ToLower(kind) /var/folders/22/T/holos.validate2549739170/components/podinfo/podinfo.gen.yaml: exit status 1 at builder/v1alpha5/builder.go:411
+could not run: could not render component: could not run command: holos --log-level info --log-format console render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo: exit status 1 at cli/render/render.go:155
+```
+
+:::important
+Holos quickly returns an error if validated artifacts have a Secret.
+:::
+
+Remove the secret to resolve the issue.
+
+```shell
+rm components/podinfo/secret.cue
+```
+
+## Inspecting the diff
+
+The validation and patch results in a correct Deployment, verified against the
+Kubernetes type definitions.
+
+```shell
+git diff
+```
+```diff
+diff --git a/deploy/components/podinfo/podinfo.gen.yaml b/deploy/components/podinfo/podinfo.gen.yaml
+index 6e4aec0..a145e3f 100644
+--- a/deploy/components/podinfo/podinfo.gen.yaml
++++ b/deploy/components/podinfo/podinfo.gen.yaml
+@@ -101,7 +101,7 @@ spec:
+           successThreshold: 1
+           timeoutSeconds: 5
+         resources:
+-          limits: null
++          limits: {}
+           requests:
+             cpu: 1m
+             memory: 16Mi
+```
+
+## Trying Locally
+
+Optionally, apply the manifests rendered by Holos to a [Local Cluster] for
+testing.
+
+[Local Cluster]: ../topics/local-cluster.mdx
+[ExternalSecret]: https://external-secrets.io/latest/api/externalsecret/
+[Artifact]: ../api/core.md#Artifact
+[BuildPlan]: ../api/core.md#BuildPlan
+[Resources]: ../api/core.md#Resources
+[Validator]: ../api/core.md#Validator
+[Transformer]: ../api/core.md#Transformer
+[Kustomize]: ../api/core.md#Kustomize
+[Generator]: ../api/core.md#Generator
+[Hello Holos]: ./hello-holos.mdx
+[cue.mod/gen/external-secrets.io/externalsecret/v1beta1/types_gen.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/cue.mod/gen/external-secrets.io/externalsecret/v1beta1/types_gen.cue#L13
+[ComponentConfig]: ../api/author.md#ComponentConfig
+[timoni]: https://timoni.sh/install/
+[resources.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/v1alpha5/resources.cue#L33

doc/website/blog/2024-11-25-validators.mdx

@@ -0,0 +1,35 @@
+---
+slug: validators-feature
+title: Validators added in Holos v0.101.0
+authors: [jeff]
+tags: [holos, feature]
+image: /img/cards/validators.png
+description: Validators are useful to enforce policy and catch Helm errors.
+---
+
+import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
+import RenderPlatformDiagram from '@site/src/diagrams/render-platform-sequence.mdx';
+
+We've added support for [Validators] in [v0.101.0].  Validators are useful to
+enforce policies and ensure consistency early in the process.  This feature
+addresses two primary use cases:
+
+1. Prevent insecure configuration early in the process.  For example, prevent
+Helm from rendering a `Secret` which would otherwise be committed to version control.
+2. Prevent unsafe configuration by validating manifests against Kubernetes core
+and custom resource type definitions.
+
+Check out the [Validators] tutorial for examples of both use cases.
+
+[Validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
+[v0.101.0]: https://github.com/holos-run/holos/releases/tag/v0.101.0
+
+{/* truncate */}
+
+<RenderingOverview />
+
+Validators complete the core functionality of the Holos manifest rendering
+pipeline.  We are seeking design partners to help enhance Generators and
+Transformers.  Validators are implemented using a generic `Command` kind, and
+we're considering a similar kind for Generators and Transformers.  Please
+connect with us if you'd like to help design these enhancements.

doc/website/src/diagrams/rendering-overview.mdx

@@ -3,21 +3,21 @@
 title: Rendering Overview
 ---
 graph LR
-    Platform[<a href="/docs/v1alpha5/api/author/#Platform">Platform</a>]
-    Component[<a href="/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
+    Platform[<a href="https://holos.run/docs/v1alpha5/api/author/#Platform">Platform</a>]
+    Component[<a href="https://holos.run/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
 
-    Helm[<a href="/docs/v1alpha5/api/author/#Helm">Helm</a>]
-    Kustomize[<a href="/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
-    Kubernetes[<a href="/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
+    Helm[<a href="https://holos.run/docs/v1alpha5/api/author/#Helm">Helm</a>]
+    Kustomize[<a href="https://holos.run/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
+    Kubernetes[<a href="https://holos.run/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
 
-    BuildPlan[<a href="/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
+    BuildPlan[<a href="https://holos.run/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
 
-    ResourcesArtifact[<a href="/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
-    GitOpsArtifact[<a href="/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
+    ResourcesArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
+    GitOpsArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
 
-    Generators[<a href="/docs/v1alpha5/api/core/#Generator">Generators</a>]
-    Transformers[<a href="/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
-    Validators[Validators]
+    Generators[<a href="https://holos.run/docs/v1alpha5/api/core/#Generator">Generators</a>]
+    Transformers[<a href="https://holos.run/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
+    Validators[<a href="https://holos.run/docs/v1alpha5/api/core/#Validator">Validators</a>]
     Files[Manifest<br/>Files]
 
     Platform --> Component

doc/website/static/_redirects

@@ -1,4 +1,14 @@
-/docs /docs/v1alpha5/ 301
-/docs/ /docs/v1alpha5/ 301
-/docs/overview /docs/v1alpha5/tutorial/overview/ 301
-/docs/overview/ /docs/v1alpha5/tutorial/overview/ 301
+/docs                /docs/v1alpha5/ 301
+/docs/               /docs/v1alpha5/ 301
+/docs/tutorial       /docs/v1alpha5/tutorial/ 301
+/docs/tutorial/      /docs/v1alpha5/tutorial/ 301
+/docs/quickstart     /docs/v1alpha5/tutorial/overview/ 301
+/docs/quickstart/    /docs/v1alpha5/tutorial/overview/ 301
+/docs/overview       /docs/v1alpha5/tutorial/overview/ 301
+/docs/overview/      /docs/v1alpha5/tutorial/overview/ 301
+/docs/topics         /docs/v1alpha5/topics/structures/ 301
+/docs/topics/        /docs/v1alpha5/topics/structures/ 301
+/docs/setup          /docs/v1alpha5/tutorial/setup/ 301
+/docs/setup/         /docs/v1alpha5/tutorial/setup/ 301
+/docs/local-cluster  /docs/v1alpha5/topics/local-cluster/ 301
+/docs/local-cluster/ /docs/v1alpha5/topics/local-cluster/ 301

internal/artifact/artifact.go

@@ -20,6 +20,7 @@ var _ Store = NewStore()
 type Store interface {
 	Get(path string) (data []byte, ok bool)
 	Set(path string, data []byte) error
+	// Save previously set path to dir preserving directories.
 	Save(dir, path string) error
 }
 

internal/builder/v1alpha5/builder.go

@@ -5,16 +5,17 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"io"
 	"log/slog"
 	"os"
 	"path/filepath"
 	"strings"
+	"sync"
 	"syscall"
 	"time"
 
 	"cuelang.org/go/cue"
 	core "github.com/holos-run/holos/api/core/v1alpha5"
-	"github.com/holos-run/holos/internal/artifact"
 	"github.com/holos-run/holos/internal/errors"
 	"github.com/holos-run/holos/internal/holos"
 	"github.com/holos-run/holos/internal/logger"
@@ -107,167 +108,84 @@ func (c *Component) Path() string {
 }
 
 var _ holos.BuildPlan = &BuildPlan{}
+var _ task = generatorTask{}
+var _ task = transformersTask{}
+var _ task = validatorTask{}
 
-// BuildPlan represents a component builder.
-type BuildPlan struct {
-	core.BuildPlan
-	Opts holos.BuildOpts
+type task interface {
+	id() string
+	run(ctx context.Context) error
 }
 
-// Build builds a BuildPlan into Artifact files.
-func (b *BuildPlan) Build(ctx context.Context) error {
-	name := b.BuildPlan.Metadata.Name
-	path := b.Opts.Path
-	log := logger.FromContext(ctx).With("name", name, "path", path)
-	msg := fmt.Sprintf("could not build %s", name)
-	if b.BuildPlan.Spec.Disabled {
-		log.WarnContext(ctx, fmt.Sprintf("%s: disabled", msg))
-		return nil
-	}
+type taskParams struct {
+	taskName      string
+	buildPlanName string
+	opts          holos.BuildOpts
+}
 
-	g, ctx := errgroup.WithContext(ctx)
-	// One more for the producer
-	g.SetLimit(b.Opts.Concurrency + 1)
+func (t taskParams) id() string {
+	return fmt.Sprintf("%s:%s/%s", t.opts.Path, t.buildPlanName, t.taskName)
+}
 
-	// Producer.
-	g.Go(func() error {
-		for _, a := range b.BuildPlan.Spec.Artifacts {
-			msg := fmt.Sprintf("%s artifact %s", msg, a.Artifact)
-			log := log.With("artifact", a.Artifact)
-			if a.Skip {
-				log.WarnContext(ctx, fmt.Sprintf("%s: skipped field is true", msg))
-				continue
-			}
-			select {
-			case <-ctx.Done():
-				return ctx.Err()
-			default:
-				// https://golang.org/doc/faq#closures_and_goroutines
-				a := a
-				// Worker.  Blocks if limit has been reached.
-				g.Go(func() error {
-					for _, gen := range a.Generators {
-						switch gen.Kind {
-						case "Resources":
-							if err := b.resources(log, gen, b.Opts.Store); err != nil {
-								return errors.Format("could not generate resources: %w", err)
-							}
-						case "Helm":
-							if err := b.helm(ctx, log, gen, b.Opts.Store); err != nil {
-								return errors.Format("could not generate helm: %w", err)
-							}
-						case "File":
-							if err := b.file(log, gen, b.Opts.Store); err != nil {
-								return errors.Format("could not generate file: %w", err)
-							}
-						default:
-							return errors.Format("%s: unsupported kind %s", msg, gen.Kind)
-						}
-					}
+type generatorTask struct {
+	taskParams
+	generator core.Generator
+	wg        *sync.WaitGroup
+}
 
-					for _, t := range a.Transformers {
-						switch t.Kind {
-						case "Kustomize":
-							if err := b.kustomize(ctx, log, t, b.Opts.Store); err != nil {
-								return errors.Wrap(err)
-							}
-						case "Join":
-							s := make([][]byte, 0, len(t.Inputs))
-							for _, input := range t.Inputs {
-								if data, ok := b.Opts.Store.Get(string(input)); ok {
-									s = append(s, data)
-								} else {
-									return errors.Format("%s: missing %s", msg, input)
-								}
-							}
-							data := bytes.Join(s, []byte(t.Join.Separator))
-							if err := b.Opts.Store.Set(string(t.Output), data); err != nil {
-								return errors.Format("%s: %w", msg, err)
-							}
-							log.Debug("set artifact: " + string(t.Output))
-						default:
-							return errors.Format("%s: unsupported kind %s", msg, t.Kind)
-						}
-					}
-
-					for _, validator := range a.Validators {
-						switch validator.Kind {
-						case "Command":
-							if err := b.validate(ctx, log, validator, b.Opts.Store); err != nil {
-								return errors.Wrap(err)
-							}
-						default:
-							return errors.Format("%s: unsupported kind %s", msg, validator.Kind)
-						}
-					}
-
-					// Write the final artifact
-					if err := b.Opts.Store.Save(b.Opts.WriteTo, string(a.Artifact)); err != nil {
-						return errors.Format("%s: %w", msg, err)
-					}
-					log.DebugContext(ctx, "wrote "+filepath.Join(b.Opts.WriteTo, string(a.Artifact)))
-
-					return nil
-				})
-			}
+func (t generatorTask) run(ctx context.Context) error {
+	defer t.wg.Done()
+	msg := fmt.Sprintf("could not build %s", t.id())
+	switch t.generator.Kind {
+	case "Resources":
+		if err := t.resources(); err != nil {
+			return errors.Format("%s: could not generate resources: %w", msg, err)
 		}
-		return nil
-	})
-
-	// Wait for completion and return the first error (if any)
-	return g.Wait()
-}
-
-func (b *BuildPlan) Export(idx int, encoder holos.OrderedEncoder) error {
-	if err := encoder.Encode(idx, &b.BuildPlan); err != nil {
-		return errors.Wrap(err)
+	case "Helm":
+		if err := t.helm(ctx); err != nil {
+			return errors.Format("%s: could not generate helm: %w", msg, err)
+		}
+	case "File":
+		if err := t.file(); err != nil {
+			return errors.Format("%s: could not generate file: %w", msg, err)
+		}
+	default:
+		return errors.Format("%s: unsupported kind %s", msg, t.generator.Kind)
 	}
 	return nil
 }
 
-func (b *BuildPlan) Load(v cue.Value) error {
-	return errors.Wrap(v.Decode(&b.BuildPlan))
-}
-
-func (b *BuildPlan) file(
-	log *slog.Logger,
-	g core.Generator,
-	store artifact.Store,
-) error {
-	data, err := os.ReadFile(filepath.Join(string(b.Opts.Path), string(g.File.Source)))
+func (t generatorTask) file() error {
+	data, err := os.ReadFile(filepath.Join(string(t.opts.Path), string(t.generator.File.Source)))
 	if err != nil {
 		return errors.Wrap(err)
 	}
-	if err := store.Set(string(g.Output), data); err != nil {
+	if err := t.opts.Store.Set(string(t.generator.Output), data); err != nil {
 		return errors.Wrap(err)
 	}
-	log.Debug("set artifact: " + string(g.Output))
 	return nil
 }
 
-func (b *BuildPlan) helm(
-	ctx context.Context,
-	log *slog.Logger,
-	g core.Generator,
-	store artifact.Store,
-) error {
-	chartName := g.Helm.Chart.Name
-	log = log.With("chart", chartName)
+func (t generatorTask) helm(ctx context.Context) error {
+	chartName := t.generator.Helm.Chart.Name
 	// Unnecessary? cargo cult copied from internal/cli/render/render.go
 	if chartName == "" {
 		return errors.New("missing chart name")
 	}
-
 	// Cache the chart by version to pull new versions. (#273)
-	cacheDir := filepath.Join(string(b.Opts.Path), "vendor", g.Helm.Chart.Version)
+	cacheDir := filepath.Join(string(t.opts.Path), "vendor", t.generator.Helm.Chart.Version)
 	cachePath := filepath.Join(cacheDir, filepath.Base(chartName))
 
+	log := logger.FromContext(ctx)
+
 	if _, err := os.Stat(cachePath); os.IsNotExist(err) {
-		timeout, cancel := context.WithTimeout(ctx, 5*time.Minute)
-		defer cancel()
-		err := onceWithLock(log, timeout, cachePath, func() error {
-			return b.cacheChart(ctx, log, cacheDir, g.Helm.Chart)
-		})
+		err := func() error {
+			ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
+			defer cancel()
+			return onceWithLock(log, ctx, cachePath, func() error {
+				return cacheChart(ctx, cacheDir, t.generator.Helm.Chart, t.opts.Stderr)
+			})
+		}()
 		if err != nil {
 			return errors.Format("could not cache chart: %w", err)
 		}
@@ -280,7 +198,7 @@ func (b *BuildPlan) helm(
 	}
 	defer util.Remove(ctx, tempDir)
 
-	data, err := yaml.Marshal(g.Helm.Values)
+	data, err := yaml.Marshal(t.generator.Helm.Values)
 	if err != nil {
 		return errors.Format("could not marshal values: %w", err)
 	}
@@ -293,22 +211,22 @@ func (b *BuildPlan) helm(
 
 	// Run charts
 	args := []string{"template"}
-	if !g.Helm.EnableHooks {
+	if !t.generator.Helm.EnableHooks {
 		args = append(args, "--no-hooks")
 	}
-	for _, apiVersion := range g.Helm.APIVersions {
+	for _, apiVersion := range t.generator.Helm.APIVersions {
 		args = append(args, "--api-versions", apiVersion)
 	}
-	if kubeVersion := g.Helm.KubeVersion; kubeVersion != "" {
+	if kubeVersion := t.generator.Helm.KubeVersion; kubeVersion != "" {
 		args = append(args, "--kube-version", kubeVersion)
 	}
 	args = append(args,
 		"--include-crds",
 		"--values", valuesPath,
-		"--namespace", g.Helm.Namespace,
+		"--namespace", t.generator.Helm.Namespace,
 		"--kubeconfig", "/dev/null",
-		"--version", g.Helm.Chart.Version,
-		g.Helm.Chart.Release,
+		"--version", t.generator.Helm.Chart.Version,
+		t.generator.Helm.Chart.Release,
 		cachePath,
 	)
 	helmOut, err := util.RunCmd(ctx, "helm", args...)
@@ -325,36 +243,31 @@ func (b *BuildPlan) helm(
 	}
 
 	// Set the artifact
-	if err := store.Set(string(g.Output), helmOut.Stdout.Bytes()); err != nil {
+	if err := t.opts.Store.Set(string(t.generator.Output), helmOut.Stdout.Bytes()); err != nil {
 		return errors.Format("could not store helm output: %w", err)
 	}
-	log.Debug("set artifact: " + string(g.Output))
+	log.Debug("set artifact: " + string(t.generator.Output))
 
 	return nil
 }
 
-func (b *BuildPlan) resources(
-	log *slog.Logger,
-	g core.Generator,
-	store artifact.Store,
-) error {
+func (t generatorTask) resources() error {
 	var size int
-	for _, m := range g.Resources {
+	for _, m := range t.generator.Resources {
 		size += len(m)
 	}
 	list := make([]core.Resource, 0, size)
 
-	for _, m := range g.Resources {
+	for _, m := range t.generator.Resources {
 		for _, r := range m {
 			list = append(list, r)
 		}
 	}
 
 	msg := fmt.Sprintf(
-		"could not generate %s for %s path %s",
-		g.Output,
-		b.BuildPlan.Metadata.Name,
-		b.Opts.Path,
+		"could not generate %s for %s",
+		t.generator.Output,
+		t.id(),
 	)
 
 	buf, err := marshal(list)
@@ -362,114 +275,220 @@ func (b *BuildPlan) resources(
 		return errors.Format("%s: %w", msg, err)
 	}
 
-	if err := store.Set(string(g.Output), buf.Bytes()); err != nil {
-		return errors.Format("%s: %w", msg, err)
-	}
-
-	log.Debug("set artifact " + string(g.Output))
-	return nil
-}
-
-func (b *BuildPlan) validate(
-	ctx context.Context,
-	log *slog.Logger,
-	validator core.Validator,
-	store artifact.Store,
-) error {
-	tempDir, err := os.MkdirTemp("", "holos.validate")
-	if err != nil {
-		return errors.Wrap(err)
-	}
-	defer util.Remove(ctx, tempDir)
-	msg := fmt.Sprintf(
-		"could not validate %s path %s",
-		b.BuildPlan.Metadata.Name,
-		b.Opts.Path,
-	)
-
-	// Write the inputs
-	for _, input := range validator.Inputs {
-		path := string(input)
-		if err := store.Save(tempDir, path); err != nil {
-			return errors.Format("%s: %w", msg, err)
-		}
-		log.DebugContext(ctx, "wrote "+filepath.Join(tempDir, path))
-	}
-
-	if len(validator.Command.Args) < 1 {
-		return errors.Format("%s: command args length must be at least 1", msg)
-	}
-	size := len(validator.Command.Args) + len(validator.Inputs)
-	args := make([]string, 0, size)
-	args = append(args, validator.Command.Args...)
-	for _, input := range validator.Inputs {
-		args = append(args, filepath.Join(tempDir, string(input)))
-	}
-
-	// Execute the validator
-	if _, err = util.RunCmdA(ctx, b.Opts.Stderr, args[0], args[1:]...); err != nil {
+	if err := t.opts.Store.Set(string(t.generator.Output), buf.Bytes()); err != nil {
 		return errors.Format("%s: %w", msg, err)
 	}
 
 	return nil
 }
 
-func (b *BuildPlan) kustomize(
-	ctx context.Context,
-	log *slog.Logger,
-	t core.Transformer,
-	store artifact.Store,
-) error {
-	tempDir, err := os.MkdirTemp("", "holos.kustomize")
-	if err != nil {
-		return errors.Wrap(err)
-	}
-	defer util.Remove(ctx, tempDir)
-	msg := fmt.Sprintf(
-		"could not transform %s for %s path %s",
-		t.Output,
-		b.BuildPlan.Metadata.Name,
-		b.Opts.Path,
-	)
+type transformersTask struct {
+	taskParams
+	transformers []core.Transformer
+	wg           *sync.WaitGroup
+}
 
-	// Write the kustomization
-	data, err := yaml.Marshal(t.Kustomize.Kustomization)
-	if err != nil {
-		return errors.Format("%s: %w", msg, err)
-	}
-	path := filepath.Join(tempDir, "kustomization.yaml")
-	if err := os.WriteFile(path, data, 0666); err != nil {
-		return errors.Format("%s: %w", msg, err)
-	}
-	log.DebugContext(ctx, "wrote "+path)
-
-	// Write the inputs
-	for _, input := range t.Inputs {
-		path := string(input)
-		if err := store.Save(tempDir, path); err != nil {
-			return errors.Format("%s: %w", msg, err)
+func (t transformersTask) run(ctx context.Context) error {
+	defer t.wg.Done()
+	for idx, transformer := range t.transformers {
+		msg := fmt.Sprintf("could not build %s/%d", t.id(), idx)
+		switch transformer.Kind {
+		case "Kustomize":
+			if err := kustomize(ctx, transformer, t.taskParams); err != nil {
+				return errors.Wrap(err)
+			}
+		case "Join":
+			s := make([][]byte, 0, len(transformer.Inputs))
+			for _, input := range transformer.Inputs {
+				if data, ok := t.opts.Store.Get(string(input)); ok {
+					s = append(s, data)
+				} else {
+					return errors.Format("%s: missing %s", msg, input)
+				}
+			}
+			data := bytes.Join(s, []byte(transformer.Join.Separator))
+			if err := t.opts.Store.Set(string(transformer.Output), data); err != nil {
+				return errors.Format("%s: %w", msg, err)
+			}
+		default:
+			return errors.Format("%s: unsupported kind %s", msg, transformer.Kind)
 		}
-		log.DebugContext(ctx, "wrote "+filepath.Join(tempDir, path))
 	}
+	return nil
+}
 
-	// Execute kustomize
-	r, err := util.RunCmdW(ctx, b.Opts.Stderr, "kubectl", "kustomize", tempDir)
-	if err != nil {
-		kErr := r.Stderr.String()
-		err = errors.Format("%s: could not run kustomize: %w", msg, err)
-		log.ErrorContext(ctx, fmt.Sprintf("%s: stderr:\n%s", err.Error(), kErr), "err", err, "stderr", kErr)
-		return err
+type validatorTask struct {
+	taskParams
+	validator core.Validator
+	wg        *sync.WaitGroup
+}
+
+func (t validatorTask) run(ctx context.Context) error {
+	defer t.wg.Done()
+	msg := fmt.Sprintf("could not validate %s", t.id())
+	switch kind := t.validator.Kind; kind {
+	case "Command":
+		if err := validate(ctx, t.validator, t.taskParams); err != nil {
+			return errors.Wrap(err)
+		}
+	default:
+		return errors.Format("%s: unsupported kind %s", msg, kind)
 	}
+	return nil
+}
 
-	// Store the artifact
-	if err := store.Set(string(t.Output), r.Stdout.Bytes()); err != nil {
+func worker(ctx context.Context, idx int, tasks chan task) error {
+	log := logger.FromContext(ctx).With("worker", idx)
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case task, ok := <-tasks:
+			if !ok {
+				log.DebugContext(ctx, fmt.Sprintf("worker %d returning: tasks chan closed", idx))
+				return nil
+			}
+			log.DebugContext(ctx, fmt.Sprintf("worker %d task %s starting", idx, task.id()))
+			if err := task.run(ctx); err != nil {
+				return errors.Wrap(err)
+			}
+			log.DebugContext(ctx, fmt.Sprintf("worker %d task %s finished ok", idx, task.id()))
+		}
+	}
+}
+
+func buildArtifact(ctx context.Context, idx int, artifact core.Artifact, tasks chan task, buildPlanName string, opts holos.BuildOpts) error {
+	var wg sync.WaitGroup
+	msg := fmt.Sprintf("could not build %s artifact %s", buildPlanName, artifact.Artifact)
+	// Process Generators concurrently
+	for gid, gen := range artifact.Generators {
+		task := generatorTask{
+			taskParams: taskParams{
+				taskName:      fmt.Sprintf("artifact/%d/generator/%d", idx, gid),
+				buildPlanName: buildPlanName,
+				opts:          opts,
+			},
+			generator: gen,
+			wg:        &wg,
+		}
+		wg.Add(1)
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case tasks <- task:
+		}
+	}
+	wg.Wait()
+
+	// Process Transformers sequentially
+	task := transformersTask{
+		taskParams: taskParams{
+			taskName:      fmt.Sprintf("artifact/%d/transformers", idx),
+			buildPlanName: buildPlanName,
+			opts:          opts,
+		},
+		transformers: artifact.Transformers,
+		wg:           &wg,
+	}
+	wg.Add(1)
+	select {
+	case <-ctx.Done():
+		return ctx.Err()
+	case tasks <- task:
+	}
+	wg.Wait()
+
+	// Process Validators concurrently
+	for vid, val := range artifact.Validators {
+		task := validatorTask{
+			taskParams: taskParams{
+				taskName:      fmt.Sprintf("artifact/%d/validator/%d", idx, vid),
+				buildPlanName: buildPlanName,
+				opts:          opts,
+			},
+			validator: val,
+			wg:        &wg,
+		}
+		wg.Add(1)
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case tasks <- task:
+		}
+	}
+	wg.Wait()
+
+	// Write the final artifact
+	out := string(artifact.Artifact)
+	if err := opts.Store.Save(opts.WriteTo, out); err != nil {
 		return errors.Format("%s: %w", msg, err)
 	}
-	log.Debug("set artifact " + string(t.Output))
+	log := logger.FromContext(ctx)
+	log.DebugContext(ctx, fmt.Sprintf("wrote %s", filepath.Join(opts.WriteTo, out)))
 
 	return nil
 }
 
+// BuildPlan represents a component builder.
+type BuildPlan struct {
+	core.BuildPlan
+	Opts holos.BuildOpts
+}
+
+func (b *BuildPlan) Build(ctx context.Context) error {
+	name := b.BuildPlan.Metadata.Name
+	path := b.Opts.Path
+	log := logger.FromContext(ctx).With("name", name, "path", path)
+
+	msg := fmt.Sprintf("could not build %s", name)
+	if b.BuildPlan.Spec.Disabled {
+		log.WarnContext(ctx, fmt.Sprintf("%s: disabled", msg))
+		return nil
+	}
+
+	g, ctx := errgroup.WithContext(ctx)
+	tasks := make(chan task)
+
+	// Start the worker pool.
+	for idx := 0; idx < max(1, b.Opts.Concurrency); idx++ {
+		g.Go(func() error {
+			return worker(ctx, idx, tasks)
+		})
+	}
+
+	// Start one producer that fans out to one pipeline per artifact.
+	g.Go(func() error {
+		// Close the tasks chan when the producer returns.
+		defer func() {
+			log.DebugContext(ctx, "producer returning: closing tasks chan")
+			close(tasks)
+		}()
+		// Separate error group for producers.
+		p, ctx := errgroup.WithContext(ctx)
+		for idx, a := range b.BuildPlan.Spec.Artifacts {
+			p.Go(func() error {
+				return buildArtifact(ctx, idx, a, tasks, b.Metadata.Name, b.Opts)
+			})
+		}
+		// Wait on producers to finish.
+		return errors.Wrap(p.Wait())
+	})
+
+	// Wait on workers to finish.
+	return g.Wait()
+}
+
+func (b *BuildPlan) Export(idx int, encoder holos.OrderedEncoder) error {
+	if err := encoder.Encode(idx, &b.BuildPlan); err != nil {
+		return errors.Wrap(err)
+	}
+	return nil
+}
+
+func (b *BuildPlan) Load(v cue.Value) error {
+	return errors.Wrap(v.Decode(&b.BuildPlan))
+}
+
 func marshal(list []core.Resource) (buf bytes.Buffer, err error) {
 	encoder := yaml.NewEncoder(&buf)
 	defer encoder.Close()
@@ -482,27 +501,59 @@ func marshal(list []core.Resource) (buf bytes.Buffer, err error) {
 	return
 }
 
-// cacheChart stores a cached copy of Chart in the chart subdirectory of path.
-//
-// We assume the only method responsible for writing to chartDir is cacheChart
-// itself.  cacheChart runs concurrently when rendering a platform.
-//
-// We rely on the atomicity of moving temporary directories into place on the
-// same filesystem via os.Rename. If a syscall.EEXIST error occurs during
-// renaming, it indicates that the cached chart already exists, which is
-// expected when this function is called concurrently.
-//
-// TODO(jeff): Break the dependency on v1alpha5, make it work across versions as
-// a utility function.
-func (b *BuildPlan) cacheChart(
-	ctx context.Context,
-	log *slog.Logger,
-	cacheDir string,
-	chart core.Chart,
-) error {
+// onceWithLock obtains a filesystem lock with mkdir, then executes fn.  If the
+// lock is already locked, onceWithLock waits for it to be released then returns
+// without calling fn.
+func onceWithLock(log *slog.Logger, ctx context.Context, path string, fn func() error) error {
+	if err := os.MkdirAll(filepath.Dir(path), 0777); err != nil {
+		return errors.Wrap(err)
+	}
+
+	// Obtain a lock with a timeout.
+	lockDir := path + ".lock"
+	log = log.With("lock", lockDir)
+
+	err := os.Mkdir(lockDir, 0777)
+	if err == nil {
+		defer os.RemoveAll(lockDir)
+		log.DebugContext(ctx, fmt.Sprintf("acquired %s", lockDir))
+		if err := fn(); err != nil {
+			return errors.Wrap(err)
+		}
+		log.DebugContext(ctx, fmt.Sprintf("released %s", lockDir))
+		return nil
+	}
+
+	// Wait until the lock is released then return.
+	if os.IsExist(err) {
+		log.DebugContext(ctx, fmt.Sprintf("blocked %s", lockDir))
+		stillBlocked := time.After(5 * time.Second)
+		deadLocked := time.After(10 * time.Second)
+		for {
+			select {
+			case <-stillBlocked:
+				log.WarnContext(ctx, fmt.Sprintf("waiting for %s to be released", lockDir))
+			case <-deadLocked:
+				log.WarnContext(ctx, fmt.Sprintf("still waiting for %s to be released (dead lock?)", lockDir))
+			case <-time.After(100 * time.Millisecond):
+				if _, err := os.Stat(lockDir); os.IsNotExist(err) {
+					log.DebugContext(ctx, fmt.Sprintf("unblocked %s", lockDir))
+					return nil
+				}
+			case <-ctx.Done():
+				return errors.Wrap(ctx.Err())
+			}
+		}
+	}
+
+	// Unexpected error
+	return errors.Wrap(err)
+}
+
+func cacheChart(ctx context.Context, cacheDir string, chart core.Chart, stderr io.Writer) error {
+	log := logger.FromContext(ctx)
 	// Add repositories
 	repo := chart.Repository
-	stderr := b.Opts.Stderr
 	if repo.URL == "" {
 		// repo update not needed for oci charts so this is debug instead of warn.
 		log.DebugContext(ctx, "skipped helm repo add and update: repo url is empty")
@@ -571,51 +622,82 @@ func (b *BuildPlan) cacheChart(
 	return nil
 }
 
-// onceWithLock obtains a filesystem lock with mkdir, then executes fn.  If the
-// lock is already locked, onceWithLock waits for it to be released then returns
-// without calling fn.
-func onceWithLock(log *slog.Logger, ctx context.Context, path string, fn func() error) error {
-	if err := os.MkdirAll(filepath.Dir(path), 0777); err != nil {
+func kustomize(ctx context.Context, t core.Transformer, p taskParams) error {
+	tempDir, err := os.MkdirTemp("", "holos.kustomize")
+	if err != nil {
 		return errors.Wrap(err)
 	}
+	defer util.Remove(ctx, tempDir)
+	msg := fmt.Sprintf(
+		"could not transform %s for %s path %s",
+		t.Output,
+		p.buildPlanName,
+		p.opts.Path,
+	)
 
-	// Obtain a lock with a timeout.
-	lockDir := path + ".lock"
-	log = log.With("lock", lockDir)
-
-	err := os.Mkdir(lockDir, 0777)
-	if err == nil {
-		defer os.RemoveAll(lockDir)
-		log.DebugContext(ctx, fmt.Sprintf("acquired %s", lockDir))
-		if err := fn(); err != nil {
-			return errors.Wrap(err)
-		}
-		log.DebugContext(ctx, fmt.Sprintf("released %s", lockDir))
-		return nil
+	// Write the kustomization
+	data, err := yaml.Marshal(t.Kustomize.Kustomization)
+	if err != nil {
+		return errors.Format("%s: %w", msg, err)
+	}
+	path := filepath.Join(tempDir, "kustomization.yaml")
+	if err := os.WriteFile(path, data, 0666); err != nil {
+		return errors.Format("%s: %w", msg, err)
 	}
 
-	// Wait until the lock is released then return.
-	if os.IsExist(err) {
-		log.DebugContext(ctx, fmt.Sprintf("blocked %s", lockDir))
-		stillBlocked := time.After(5 * time.Second)
-		deadLocked := time.After(10 * time.Second)
-		for {
-			select {
-			case <-stillBlocked:
-				log.WarnContext(ctx, fmt.Sprintf("waiting for %s to be released", lockDir))
-			case <-deadLocked:
-				log.WarnContext(ctx, fmt.Sprintf("still waiting for %s to be released (dead lock?)", lockDir))
-			case <-time.After(100 * time.Millisecond):
-				if _, err := os.Stat(lockDir); os.IsNotExist(err) {
-					log.DebugContext(ctx, fmt.Sprintf("unblocked %s", lockDir))
-					return nil
-				}
-			case <-ctx.Done():
-				return errors.Wrap(ctx.Err())
-			}
+	// Write the inputs
+	for _, input := range t.Inputs {
+		path := string(input)
+		if err := p.opts.Store.Save(tempDir, path); err != nil {
+			return errors.Format("%s: %w", msg, err)
 		}
 	}
 
-	// Unexpected error
-	return errors.Wrap(err)
+	// Execute kustomize
+	r, err := util.RunCmdW(ctx, p.opts.Stderr, "kubectl", "kustomize", tempDir)
+	if err != nil {
+		return errors.Format("%s: could not run kustomize: %w", msg, err)
+	}
+
+	// Store the artifact
+	if err := p.opts.Store.Set(string(t.Output), r.Stdout.Bytes()); err != nil {
+		return errors.Format("%s: %w", msg, err)
+	}
+
+	return nil
+}
+
+func validate(ctx context.Context, validator core.Validator, p taskParams) error {
+	store := p.opts.Store
+	tempDir, err := os.MkdirTemp("", "holos.validate")
+	if err != nil {
+		return errors.Wrap(err)
+	}
+	// defer util.Remove(ctx, tempDir)
+	msg := fmt.Sprintf("could not validate %s", p.id())
+
+	// Write the inputs
+	for _, input := range validator.Inputs {
+		path := string(input)
+		if err := store.Save(tempDir, path); err != nil {
+			return errors.Format("%s: %w", msg, err)
+		}
+	}
+
+	if len(validator.Command.Args) < 1 {
+		return errors.Format("%s: command args length must be at least 1", msg)
+	}
+	size := len(validator.Command.Args) + len(validator.Inputs)
+	args := make([]string, 0, size)
+	args = append(args, validator.Command.Args...)
+	for _, input := range validator.Inputs {
+		args = append(args, filepath.Join(tempDir, string(input)))
+	}
+
+	// Execute the validator
+	if _, err = util.RunCmdA(ctx, p.opts.Stderr, args[0], args[1:]...); err != nil {
+		return errors.Format("%s: %w", msg, err)
+	}
+
+	return nil
 }

internal/cli/main.go

@@ -4,6 +4,9 @@ import (
 	"context"
 	"fmt"
 	"log/slog"
+	"os"
+	"runtime/pprof"
+	"runtime/trace"
 
 	"connectrpc.com/connect"
 	cue "cuelang.org/go/cue/errors"
@@ -12,12 +15,48 @@ import (
 	"google.golang.org/genproto/googleapis/rpc/errdetails"
 )
 
+func memProfile(ctx context.Context, cfg *holos.Config) {
+	if format := os.Getenv("HOLOS_MEM_PROFILE"); format != "" {
+		f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
+		defer f.Close()
+		if err := pprof.WriteHeapProfile(f); err != nil {
+			_ = HandleError(ctx, err, cfg)
+		}
+	}
+}
+
 // MakeMain makes a main function for the cli or tests.
 func MakeMain(options ...holos.Option) func() int {
 	return func() (exitCode int) {
 		cfg := holos.New(options...)
 		slog.SetDefault(cfg.Logger())
 		ctx := context.Background()
+
+		if format := os.Getenv("HOLOS_CPU_PROFILE"); format != "" {
+			f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
+			err := pprof.StartCPUProfile(f)
+			defer func() {
+				pprof.StopCPUProfile()
+				f.Close()
+			}()
+			if err != nil {
+				return HandleError(ctx, err, cfg)
+			}
+		}
+		defer memProfile(ctx, cfg)
+
+		if format := os.Getenv("HOLOS_TRACE"); format != "" {
+			f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
+			err := trace.Start(f)
+			defer func() {
+				trace.Stop()
+				f.Close()
+			}()
+			if err != nil {
+				return HandleError(ctx, err, cfg)
+			}
+		}
+
 		feature := &holos.EnvFlagger{}
 		if err := New(cfg, feature).ExecuteContext(ctx); err != nil {
 			return HandleError(ctx, err, cfg)

internal/generate/platforms/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha5/types_go_gen.cue

@@ -232,7 +232,10 @@ package core
 #FileContent: string
 
 // Validator validates files.  Useful to validate an [Artifact] prior to writing
-// it out to the final destination.  Validators may be executed concurrently.
+// it out to the final destination.  Holos may execute validators concurrently.
+// See the [validators] tutorial for an end to end example.
+//
+// [validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
 #Validator: {
 	// Kind represents the kind of transformer. Must be Kustomize, or Join.
 	kind: string & "Command" @go(Kind)

version/embedded/patch

@@ -1 +1 @@
-0
+1