mirror of
https://github.com/holos-run/holos.git
synced 2026-03-19 16:54:58 +00:00
Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
16a6447926 | ||
|
|
111a5944ff | ||
|
|
ff1446dc93 | ||
|
|
67ef990c37 | ||
|
|
6bd54ab856 | ||
|
|
89a23a10fd | ||
|
|
5a939bb6fe | ||
|
|
ee16f14e03 | ||
|
|
7530345620 | ||
|
|
47d60ef86d | ||
|
|
9e9f6efd04 | ||
|
|
fb4a043823 | ||
|
|
d718ab1910 | ||
|
|
c649db18a9 | ||
|
|
b3bddf3ee3 | ||
|
|
77836be250 | ||
|
|
4db670b854 | ||
|
|
d87c919519 |
129
README.md
129
README.md
@@ -1,35 +1,130 @@
|
||||
## Holos - A Holistic Development Platform
|
||||
# Holos
|
||||
|
||||
<img width="50%"
|
||||
align="right"
|
||||
style="display: block; margin: 40px auto;"
|
||||
src="https://openinfrastructure.co/blog/2016/02/27/logo/logorectangle.png">
|
||||
|
||||
Building and maintaining a software development platform is a complex and time
|
||||
consuming endeavour. Organizations often dedicate a team of 3-4 who need 6-12
|
||||
months to build the platform.
|
||||
[Holos] is a configuration management tool for Kubernetes implementing the
|
||||
[rendered manifests pattern]. It handles configurations ranging from single
|
||||
resources to multi-cluster platforms across regions.
|
||||
|
||||
Holos is a tool and a reference platform to reduce the complexity and speed up
|
||||
the process of building a modern, cloud native software development platform.
|
||||
Key components:
|
||||
- Platform schemas defining component integration
|
||||
- Building blocks unifying Helm, Kustomize and Kubernetes configs with CUE
|
||||
- BuildPlan pipeline for generating, transforming and validating manifests
|
||||
|
||||
- **Accelerate new projects** - Reduce time to market and operational complexity by starting your new project on top of the Holos reference platform.
|
||||
- **Modernize existing projects** - Incrementally incorporate your existing platform services into Holos for simpler integration.
|
||||
- **Unified configuration model** - Increase safety and reduce the risk of config changes with CUE.
|
||||
- **First class Helm and Kustomize support** - Leverage and reuse your existing investment in existing configuration tools such as Helm and Kustomize.
|
||||
- **Modern Authentication and Authorization** - Holos seamlessly integrates platform identity and access management with zero-trust beyond corp style authorization policy.
|
||||
```mermaid
|
||||
---
|
||||
title: Rendering Overview
|
||||
---
|
||||
graph LR
|
||||
Platform[<a href="https://holos.run/docs/v1alpha5/api/author/#Platform">Platform</a>]
|
||||
Component[<a href="https://holos.run/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
|
||||
|
||||
## Quick Installation
|
||||
Helm[<a href="https://holos.run/docs/v1alpha5/api/author/#Helm">Helm</a>]
|
||||
Kustomize[<a href="https://holos.run/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
|
||||
Kubernetes[<a href="https://holos.run/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
|
||||
|
||||
```console
|
||||
go install github.com/holos-run/holos/cmd/holos@latest
|
||||
BuildPlan[<a href="https://holos.run/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
|
||||
|
||||
ResourcesArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
|
||||
GitOpsArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
|
||||
|
||||
Generators[<a href="https://holos.run/docs/v1alpha5/api/core/#Generator">Generators</a>]
|
||||
Transformers[<a href="https://holos.run/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
|
||||
Validators[<a href="https://holos.run/docs/v1alpha5/api/core/#Validator">Validators</a>]
|
||||
Files[Manifest<br/>Files]
|
||||
|
||||
Platform --> Component
|
||||
Component --> Helm --> BuildPlan
|
||||
Component --> Kubernetes --> BuildPlan
|
||||
Component --> Kustomize --> BuildPlan
|
||||
|
||||
BuildPlan --> ResourcesArtifact --> Generators
|
||||
BuildPlan --> GitOpsArtifact --> Generators
|
||||
|
||||
Generators --> Transformers --> Validators --> Files
|
||||
```
|
||||
|
||||
## Docs and Support
|
||||
## Setup
|
||||
|
||||
The documentation for developing and using Holos is available at: https://holos.run
|
||||
```shell
|
||||
brew install holos-run/tap/holos
|
||||
```
|
||||
|
||||
For discussion and support, [open a discussion](https://github.com/orgs/holos-run/discussions/new/choose).
|
||||
Refer to [setup] for other installation methods and dependencies.
|
||||
|
||||
## Example
|
||||
|
||||
See our [tutorial] for a complete hello world example.
|
||||
|
||||
```cue showLineNumbers
|
||||
package holos
|
||||
|
||||
holos: Component.BuildPlan
|
||||
|
||||
Component: #Helm & {
|
||||
Name: "podinfo"
|
||||
Chart: {
|
||||
version: "6.6.2"
|
||||
repository: {
|
||||
name: "podinfo"
|
||||
url: "https://stefanprodan.github.io/podinfo"
|
||||
}
|
||||
}
|
||||
Values: ui: {
|
||||
message: string | *"Hello World" @tag(message, type=string)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Organizational Role
|
||||
|
||||
Platform engineers use Holos to generate Kubernetes manifests, both locally and
|
||||
in CI pipelines. The manifests are committed to version control and deployed via
|
||||
GitOps tools like ArgoCD or Flux.
|
||||
|
||||
Holos integrates seamlessly with existing Helm charts, Kustomize bases, and
|
||||
other version-controlled configurations.
|
||||
|
||||
## Advantages of Holos
|
||||
|
||||
### Safe
|
||||
|
||||
Holos leverages [CUE] for strong typing and validation of configuration data,
|
||||
ensuring consistent output from Helm and other tools.
|
||||
|
||||
### Consistent
|
||||
|
||||
A unified pipeline processes all configurations - whether from CUE, Helm, or
|
||||
Kustomize - through the same well-defined stages.
|
||||
|
||||
### Flexible
|
||||
|
||||
Composable building blocks for generation, transformation, validation and
|
||||
integration let teams assemble workflows that match their needs.
|
||||
|
||||
The core is intentionally unopinionated about platform configuration patterns.
|
||||
Common needs like environments and clusters are provided as customizable
|
||||
[topics] recipes rather than enforced structures.
|
||||
|
||||
## Getting Help
|
||||
|
||||
Get support through our [Discord] channel or [GitHub discussions]. Configuration
|
||||
challenges arise at all experience levels - we welcome your questions and are
|
||||
here to help.
|
||||
|
||||
## License
|
||||
|
||||
Holos is licensed under Apache 2.0 as found in the [LICENSE file](LICENSE).
|
||||
|
||||
[Holos]: https://holos.run
|
||||
[rendered manifests pattern]: https://akuity.io/blog/the-rendered-manifests-pattern
|
||||
[CUE]: https://cuelang.org/
|
||||
[Discord]: https://discord.gg/JgDVbNpye7
|
||||
[GitHub discussions]: https://github.com/holos-run/holos/discussions
|
||||
[Why CUE for Configuration]: https://holos.run/blog/why-cue-for-configuration/
|
||||
[topics]: https://holos.run/docs/topics/
|
||||
[setup]: https://holos.run/docs/setup/
|
||||
[tutorial]: https://holos.run/docs/tutorial/
|
||||
|
||||
@@ -209,7 +209,10 @@ type FilePath string
|
||||
type FileContent string
|
||||
|
||||
// Validator validates files. Useful to validate an [Artifact] prior to writing
|
||||
// it out to the final destination. Validators may be executed concurrently.
|
||||
// it out to the final destination. Holos may execute validators concurrently.
|
||||
// See the [validators] tutorial for an end to end example.
|
||||
//
|
||||
// [validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
|
||||
type Validator struct {
|
||||
// Kind represents the kind of transformer. Must be Kustomize, or Join.
|
||||
Kind string `json:"kind" yaml:"kind" cue:"\"Command\""`
|
||||
|
||||
@@ -7,11 +7,11 @@ cd $WORK
|
||||
exec holos generate platform v1alpha5 --force
|
||||
|
||||
# Platforms are empty by default.
|
||||
exec holos render platform ./platform
|
||||
exec holos render platform
|
||||
stderr -count=1 '^rendered platform'
|
||||
|
||||
# Holos uses CUE to build a platform specification.
|
||||
exec cue export --expression holos --out=yaml ./platform
|
||||
exec holos show platform
|
||||
cmp stdout want/1.platform_spec.yaml
|
||||
|
||||
# Define the host and port in projects/blackbox.schema.cue
|
||||
@@ -22,7 +22,7 @@ mv projects/platform/components/prometheus/prometheus.cue.disabled projects/plat
|
||||
mv platform/prometheus.cue.disabled platform/prometheus.cue
|
||||
|
||||
# Render the platform to render the prometheus chart.
|
||||
exec holos render platform ./platform
|
||||
exec holos render platform
|
||||
stderr -count=1 '^rendered prometheus'
|
||||
stderr -count=1 '^rendered platform'
|
||||
cmp deploy/components/prometheus/prometheus.gen.yaml want/1.prometheus.gen.yaml
|
||||
@@ -73,8 +73,8 @@ core.#BuildPlan & {
|
||||
metadata: name: _Tags.component.name
|
||||
}
|
||||
-- want/1.platform_spec.yaml --
|
||||
kind: Platform
|
||||
apiVersion: v1alpha5
|
||||
kind: Platform
|
||||
metadata:
|
||||
name: default
|
||||
spec:
|
||||
|
||||
50
cmd/holos/tests/v1alpha5/issues/issue366-cue-build-tags.txt
Normal file
50
cmd/holos/tests/v1alpha5/issues/issue366-cue-build-tags.txt
Normal file
@@ -0,0 +1,50 @@
|
||||
# https://github.com/holos-run/holos/issues/366
|
||||
# Build tags conditionally include CUE files.
|
||||
env HOME=$WORK
|
||||
|
||||
exec holos init platform v1alpha5 --force
|
||||
exec holos show platform
|
||||
cmp stdout want/empty.yaml
|
||||
|
||||
exec holos show platform -t foo
|
||||
cmp stdout want/foo.yaml
|
||||
|
||||
-- platform/empty.cue --
|
||||
@if(foo)
|
||||
package holos
|
||||
|
||||
Platform: Components: foo: _
|
||||
-- platform/metadata.cue --
|
||||
package holos
|
||||
|
||||
Platform: Components: [NAME=string]: {
|
||||
name: NAME
|
||||
path: "components/empty"
|
||||
labels: "app.holos.run/name": NAME
|
||||
annotations: "app.holos.run/description": "\(NAME) empty test case"
|
||||
}
|
||||
-- components/empty/empty.cue --
|
||||
package holos
|
||||
|
||||
Component: #Kubernetes & {}
|
||||
holos: Component.BuildPlan
|
||||
-- want/empty.yaml --
|
||||
apiVersion: v1alpha5
|
||||
kind: Platform
|
||||
metadata:
|
||||
name: default
|
||||
spec:
|
||||
components: []
|
||||
-- want/foo.yaml --
|
||||
apiVersion: v1alpha5
|
||||
kind: Platform
|
||||
metadata:
|
||||
name: default
|
||||
spec:
|
||||
components:
|
||||
- annotations:
|
||||
app.holos.run/description: foo empty test case
|
||||
labels:
|
||||
app.holos.run/name: foo
|
||||
name: foo
|
||||
path: components/empty
|
||||
@@ -418,7 +418,7 @@ type Transformer struct {
|
||||
<a name="Validator"></a>
|
||||
## type Validator {#Validator}
|
||||
|
||||
Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Validators may be executed concurrently.
|
||||
Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Holos may execute validators concurrently. See the [validators](<https://holos.run/docs/v1alpha5/tutorial/validators/>) tutorial for an end to end example.
|
||||
|
||||
```go
|
||||
type Validator struct {
|
||||
|
||||
@@ -265,9 +265,6 @@ This tutorial uses the `#Resources` structure to map resource kinds to their
|
||||
schema definitions in CUE. This structure is defined in `resources.cue` at the
|
||||
root of the tree. Take a look at [resources.cue] to see this mapping structure.
|
||||
|
||||
Continue to the next tutorial to learn how to define your own data structures
|
||||
similar to the `#Resources` structure.
|
||||
|
||||
[Local Cluster]: ../topics/local-cluster.mdx
|
||||
[ExternalSecret]: https://external-secrets.io/latest/api/externalsecret/
|
||||
[Artifact]: ../api/core.md#Artifact
|
||||
|
||||
@@ -11,7 +11,7 @@ import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
|
||||
|
||||
## Overview
|
||||
|
||||
Holos is a configuration management tool for Kubernetes that implements the
|
||||
Holos is a configuration management tool for Kubernetes implementing the
|
||||
[rendered manifests pattern]. It handles configurations ranging from single
|
||||
resources to multi-cluster platforms across regions.
|
||||
|
||||
|
||||
428
doc/md/tutorial/validators.mdx
Normal file
428
doc/md/tutorial/validators.mdx
Normal file
@@ -0,0 +1,428 @@
|
||||
---
|
||||
slug: validators
|
||||
title: Validators
|
||||
description: Validate rendered manifests against policy definitions.
|
||||
sidebar_position: 60
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
|
||||
|
||||
# Validators
|
||||
|
||||
## Overview
|
||||
|
||||
Sometimes Helm charts render Secrets we do not wanted committed to version
|
||||
control for security. Helm charts often render incorrect manifests, even if
|
||||
they're accepted by the api server. For example, passing `null` to collection
|
||||
fields. We'll solve both of these issues using a [Validator] to block artifacts
|
||||
with a Secret resource, and verifying the artifact against Kubernetes type
|
||||
definitions.
|
||||
|
||||
1. If a Helm chart renders a Secret, Holos errors before writing the artifact
|
||||
and suggests an ExternalSecret instead.
|
||||
2. Each resource is validated against a field named by the value of the kind
|
||||
field. For example, a `kind: Secret` resource validates against `secret: {}` in
|
||||
CUE. `kind: Deployment` validates against `deployment: {}` in CUE.
|
||||
3. The final artifact is validated, covering the output of all generators and
|
||||
transformers.
|
||||
|
||||
<RenderingOverview />
|
||||
|
||||
## The Code
|
||||
|
||||
### Generating the Structure
|
||||
|
||||
Use `holos` to generate a minimal platform directory structure. First, create
|
||||
and navigate into a blank directory. Then, use the `holos generate platform`
|
||||
command to generate a minimal platform.
|
||||
|
||||
```shell
|
||||
mkdir holos-validators-tutorial && cd holos-validators-tutorial
|
||||
holos init platform v1alpha5
|
||||
```
|
||||
|
||||
### Creating the Component
|
||||
|
||||
Create the directory for the `podinfo` component. Create an empty file, then add
|
||||
the following CUE configuration to it.
|
||||
|
||||
```bash
|
||||
mkdir -p components/podinfo
|
||||
```
|
||||
```bash
|
||||
cat <<EOF > components/podinfo/podinfo.cue
|
||||
```
|
||||
```cue showLineNumbers
|
||||
package holos
|
||||
|
||||
// export the component build plan to holos
|
||||
holos: Component.BuildPlan
|
||||
|
||||
// Component is a Helm chart
|
||||
Component: #Helm & {
|
||||
Name: "podinfo"
|
||||
Namespace: "default"
|
||||
// Add metadata.namespace to all resources with kustomize.
|
||||
KustomizeConfig: Kustomization: namespace: Namespace
|
||||
Chart: {
|
||||
version: "6.6.2"
|
||||
repository: {
|
||||
name: "podinfo"
|
||||
url: "https://stefanprodan.github.io/podinfo"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
```bash
|
||||
EOF
|
||||
```
|
||||
|
||||
Register the component with the platform.
|
||||
|
||||
```bash
|
||||
cat <<EOF > platform/podinfo.cue
|
||||
```
|
||||
```cue showLineNumbers
|
||||
package holos
|
||||
|
||||
Platform: Components: podinfo: {
|
||||
name: "podinfo"
|
||||
path: "components/podinfo"
|
||||
}
|
||||
```
|
||||
```bash
|
||||
EOF
|
||||
```
|
||||
|
||||
Render the platform.
|
||||
|
||||
<Tabs groupId="tutorial-hello-render-manifests">
|
||||
<TabItem value="command" label="Command">
|
||||
```bash
|
||||
holos render platform
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="output" label="Output">
|
||||
```
|
||||
cached podinfo 6.6.2
|
||||
rendered podinfo in 1.938665041s
|
||||
rendered platform in 1.938759417s
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
Add and commit the initial configuration.
|
||||
|
||||
```bash
|
||||
git init . && git add . && git commit -m initial
|
||||
```
|
||||
|
||||
### Define the Valid Schema
|
||||
|
||||
We'll use a CUE package named `policy` so the entire platform configuration in
|
||||
package `holos` isn't loaded every time we validate an artifact.
|
||||
|
||||
Create `policy/validation-schema.cue` with the following content.
|
||||
|
||||
```shell
|
||||
mkdir -p policy
|
||||
cat <<EOF > policy/validation-schema.cue
|
||||
```
|
||||
```cue showLineNumbers
|
||||
package policy
|
||||
|
||||
import apps "k8s.io/api/apps/v1"
|
||||
|
||||
// Organize by kind then name to avoid conflicts.
|
||||
kind: [KIND=string]: [NAME=string]: {...}
|
||||
|
||||
// Useful when one component manages the same resource kind and name across
|
||||
// multiple namespaces.
|
||||
let KIND = kind
|
||||
namespace: [NS=string]: KIND
|
||||
|
||||
// Block Secret resources. kind will not unify with "Secret"
|
||||
kind: secret: [NAME=string]: kind: "Use an ExternalSecret instead. Forbidden by security policy. secret/\(NAME)"
|
||||
|
||||
// Validate Deployment against Kubernetes type definitions.
|
||||
kind: deployment: [_]: apps.#Deployment
|
||||
```
|
||||
```shell
|
||||
EOF
|
||||
```
|
||||
|
||||
### Configuring Validators
|
||||
|
||||
Configure the Validators [ComponentConfig] field to configure each [BuildPlan]
|
||||
to validate the rendered [Artifact] files.
|
||||
|
||||
```shell
|
||||
cat <<EOF > validators.cue
|
||||
```
|
||||
```cue showLineNumbers
|
||||
package holos
|
||||
|
||||
// Configure all component kinds to validate against the policy directory.
|
||||
#ComponentConfig: Validators: cue: {
|
||||
kind: "Command"
|
||||
// Note --path maps each resource to a top level field named by the kind.
|
||||
command: args: [
|
||||
"holos",
|
||||
"cue",
|
||||
"vet",
|
||||
"./policy",
|
||||
"--path=\"namespace\"",
|
||||
"--path=metadata.namespace",
|
||||
"--path=strings.ToLower(kind)",
|
||||
"--path=metadata.name",
|
||||
]
|
||||
}
|
||||
```
|
||||
```shell
|
||||
EOF
|
||||
```
|
||||
|
||||
### Patching Errors
|
||||
|
||||
Render the platform to see validation fail. The podinfo chart has no Secret,
|
||||
but it produces an invalid Deployment because it sets the container resource
|
||||
limits field to `null`.
|
||||
|
||||
```shell
|
||||
holos render platform
|
||||
```
|
||||
|
||||
```txt
|
||||
deployment.spec.template.spec.containers.0.resources.limits: conflicting values null and {[string]:"k8s.io/apimachinery/pkg/api/resource".#Quantity} (mismatched types null and struct):
|
||||
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:355:9
|
||||
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:376:12
|
||||
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2840:11
|
||||
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2968:14
|
||||
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:15
|
||||
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:18
|
||||
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:5027:9
|
||||
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:6407:16
|
||||
./policy/validation-schema.cue:9:13
|
||||
../../../../../var/folders/22/T/holos.validate1636392304/components/podinfo/podinfo.gen.yaml:104:19
|
||||
could not run: terminating because of errors
|
||||
could not run: could not validate podinfo path ./components/podinfo: could not run command: holos cue vet ./policy --path strings.ToLower(kind) /var/folders/22/T/holos.validate1636392304/components/podinfo/podinfo.gen.yaml: exit status 1 at builder/v1alpha5/builder.go:411
|
||||
could not run: could not render component: could not run command: holos --log-level info --log-format console render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo: exit status 1 at cli/render/render.go:155
|
||||
```
|
||||
|
||||
We'll use a [Kustomize] patch [Transformer] to replace the `null` limits field
|
||||
with a valid equivalent value.
|
||||
|
||||
:::important
|
||||
This configuration is defined in CUE, not YAML, even though we're configuring a
|
||||
Kustomize patch transformer. CUE gives us access to the unified platform
|
||||
configuration.
|
||||
:::
|
||||
|
||||
```shell
|
||||
cat <<EOF > components/podinfo/patch.cue
|
||||
```
|
||||
```cue showLineNumbers
|
||||
package holos
|
||||
|
||||
import "encoding/yaml"
|
||||
|
||||
Component: KustomizeConfig: Kustomization: {
|
||||
_patches: limits: {
|
||||
target: kind: "Deployment"
|
||||
patch: yaml.Marshal([{
|
||||
op: "test"
|
||||
path: "/spec/template/spec/containers/0/resources/limits"
|
||||
value: null
|
||||
}, {
|
||||
op: "replace"
|
||||
path: "/spec/template/spec/containers/0/resources/limits"
|
||||
value: {}
|
||||
}])
|
||||
}
|
||||
patches: [for x in _patches {x}]
|
||||
}
|
||||
```
|
||||
```shell
|
||||
EOF
|
||||
```
|
||||
|
||||
Now the platform renders.
|
||||
|
||||
<Tabs groupId="3A050092-8E56-49D4-84A9-71E544A21276">
|
||||
<TabItem value="command" label="Command">
|
||||
```bash
|
||||
holos render platform
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="output" label="Output">
|
||||
```txt
|
||||
rendered podinfo in 181.875083ms
|
||||
rendered platform in 181.975833ms
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Inspecting the BuildPlan
|
||||
|
||||
The BuildPlan patches the output of the upstream helm chart without modifying
|
||||
it, then validates the artifact against the Kubernetes type definitions.
|
||||
|
||||
<Tabs groupId="1DAB4C46-0793-4CCA-8930-7B2E60BDA1BE">
|
||||
<TabItem value="command" label="Command">
|
||||
```bash
|
||||
holos show buildplans
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="output" label="Output">
|
||||
```yaml showLineNumbers
|
||||
kind: BuildPlan
|
||||
apiVersion: v1alpha5
|
||||
metadata:
|
||||
name: podinfo
|
||||
spec:
|
||||
artifacts:
|
||||
- artifact: components/podinfo/podinfo.gen.yaml
|
||||
generators:
|
||||
- kind: Helm
|
||||
output: helm.gen.yaml
|
||||
helm:
|
||||
chart:
|
||||
name: podinfo
|
||||
version: 6.6.2
|
||||
release: podinfo
|
||||
repository:
|
||||
name: podinfo
|
||||
url: https://stefanprodan.github.io/podinfo
|
||||
values: {}
|
||||
namespace: default
|
||||
- kind: Resources
|
||||
output: resources.gen.yaml
|
||||
transformers:
|
||||
- kind: Kustomize
|
||||
inputs:
|
||||
- helm.gen.yaml
|
||||
- resources.gen.yaml
|
||||
output: components/podinfo/podinfo.gen.yaml
|
||||
kustomize:
|
||||
kustomization:
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: default
|
||||
patches:
|
||||
- patch: |
|
||||
- op: test
|
||||
path: /spec/template/spec/containers/0/resources/limits
|
||||
value: null
|
||||
- op: replace
|
||||
path: /spec/template/spec/containers/0/resources/limits
|
||||
value: {}
|
||||
target:
|
||||
kind: Deployment
|
||||
name: ""
|
||||
resources:
|
||||
- helm.gen.yaml
|
||||
- resources.gen.yaml
|
||||
validators:
|
||||
- kind: Command
|
||||
inputs:
|
||||
- components/podinfo/podinfo.gen.yaml
|
||||
command:
|
||||
args:
|
||||
- holos
|
||||
- cue
|
||||
- vet
|
||||
- ./policy
|
||||
- --path
|
||||
- strings.ToLower(kind)
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Catching Mistakes
|
||||
|
||||
Suppose a teammate downloads a helm chart that includes a Secret unbeknown to
|
||||
them. Holos catches the problem and suggests an ExternalSecret instead.
|
||||
|
||||
Mix in a Secret to see what happens
|
||||
|
||||
```shell
|
||||
cat <<EOF > components/podinfo/secret.cue
|
||||
```
|
||||
```cue showLineNumbers
|
||||
package holos
|
||||
|
||||
Component: Resources: Secret: example: metadata: name: "example"
|
||||
```
|
||||
```shell
|
||||
EOF
|
||||
```
|
||||
|
||||
Render the platform to see the error.
|
||||
|
||||
```shell
|
||||
holos render platform
|
||||
```
|
||||
```txt
|
||||
secret.kind: conflicting values "Use an ExternalSecret instead. Forbidden by security policy." and "Secret":
|
||||
./policy/validation-schema.cue:6:15
|
||||
../../../../../var/folders/22/T/holos.validate2549739170/components/podinfo/podinfo.gen.yaml:1:7
|
||||
could not run: terminating because of errors
|
||||
could not run: could not validate podinfo path ./components/podinfo: could not run command: holos cue vet ./policy --path strings.ToLower(kind) /var/folders/22/T/holos.validate2549739170/components/podinfo/podinfo.gen.yaml: exit status 1 at builder/v1alpha5/builder.go:411
|
||||
could not run: could not render component: could not run command: holos --log-level info --log-format console render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo: exit status 1 at cli/render/render.go:155
|
||||
```
|
||||
|
||||
:::important
|
||||
Holos quickly returns an error if validated artifacts have a Secret.
|
||||
:::
|
||||
|
||||
Remove the secret to resolve the issue.
|
||||
|
||||
```shell
|
||||
rm components/podinfo/secret.cue
|
||||
```
|
||||
|
||||
## Inspecting the diff
|
||||
|
||||
The validation and patch results in a correct Deployment, verified against the
|
||||
Kubernetes type definitions.
|
||||
|
||||
```shell
|
||||
git diff
|
||||
```
|
||||
```diff
|
||||
diff --git a/deploy/components/podinfo/podinfo.gen.yaml b/deploy/components/podinfo/podinfo.gen.yaml
|
||||
index 6e4aec0..a145e3f 100644
|
||||
--- a/deploy/components/podinfo/podinfo.gen.yaml
|
||||
+++ b/deploy/components/podinfo/podinfo.gen.yaml
|
||||
@@ -101,7 +101,7 @@ spec:
|
||||
successThreshold: 1
|
||||
timeoutSeconds: 5
|
||||
resources:
|
||||
- limits: null
|
||||
+ limits: {}
|
||||
requests:
|
||||
cpu: 1m
|
||||
memory: 16Mi
|
||||
```
|
||||
|
||||
## Trying Locally
|
||||
|
||||
Optionally, apply the manifests rendered by Holos to a [Local Cluster] for
|
||||
testing.
|
||||
|
||||
[Local Cluster]: ../topics/local-cluster.mdx
|
||||
[ExternalSecret]: https://external-secrets.io/latest/api/externalsecret/
|
||||
[Artifact]: ../api/core.md#Artifact
|
||||
[BuildPlan]: ../api/core.md#BuildPlan
|
||||
[Resources]: ../api/core.md#Resources
|
||||
[Validator]: ../api/core.md#Validator
|
||||
[Transformer]: ../api/core.md#Transformer
|
||||
[Kustomize]: ../api/core.md#Kustomize
|
||||
[Generator]: ../api/core.md#Generator
|
||||
[Hello Holos]: ./hello-holos.mdx
|
||||
[cue.mod/gen/external-secrets.io/externalsecret/v1beta1/types_gen.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/cue.mod/gen/external-secrets.io/externalsecret/v1beta1/types_gen.cue#L13
|
||||
[ComponentConfig]: ../api/author.md#ComponentConfig
|
||||
[timoni]: https://timoni.sh/install/
|
||||
[resources.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/v1alpha5/resources.cue#L33
|
||||
35
doc/website/blog/2024-11-25-validators.mdx
Normal file
35
doc/website/blog/2024-11-25-validators.mdx
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
slug: validators-feature
|
||||
title: Validators added in Holos v0.101.0
|
||||
authors: [jeff]
|
||||
tags: [holos, feature]
|
||||
image: /img/cards/validators.png
|
||||
description: Validators are useful to enforce policy and catch Helm errors.
|
||||
---
|
||||
|
||||
import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
|
||||
import RenderPlatformDiagram from '@site/src/diagrams/render-platform-sequence.mdx';
|
||||
|
||||
We've added support for [Validators] in [v0.101.0]. Validators are useful to
|
||||
enforce policies and ensure consistency early in the process. This feature
|
||||
addresses two primary use cases:
|
||||
|
||||
1. Prevent insecure configuration early in the process. For example, prevent
|
||||
Helm from rendering a `Secret` which would otherwise be committed to version control.
|
||||
2. Prevent unsafe configuration by validating manifests against Kubernetes core
|
||||
and custom resource type definitions.
|
||||
|
||||
Check out the [Validators] tutorial for examples of both use cases.
|
||||
|
||||
[Validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
|
||||
[v0.101.0]: https://github.com/holos-run/holos/releases/tag/v0.101.0
|
||||
|
||||
{/* truncate */}
|
||||
|
||||
<RenderingOverview />
|
||||
|
||||
Validators complete the core functionality of the Holos manifest rendering
|
||||
pipeline. We are seeking design partners to help enhance Generators and
|
||||
Transformers. Validators are implemented using a generic `Command` kind, and
|
||||
we're considering a similar kind for Generators and Transformers. Please
|
||||
connect with us if you'd like to help design these enhancements.
|
||||
@@ -3,21 +3,21 @@
|
||||
title: Rendering Overview
|
||||
---
|
||||
graph LR
|
||||
Platform[<a href="/docs/v1alpha5/api/author/#Platform">Platform</a>]
|
||||
Component[<a href="/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
|
||||
Platform[<a href="https://holos.run/docs/v1alpha5/api/author/#Platform">Platform</a>]
|
||||
Component[<a href="https://holos.run/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
|
||||
|
||||
Helm[<a href="/docs/v1alpha5/api/author/#Helm">Helm</a>]
|
||||
Kustomize[<a href="/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
|
||||
Kubernetes[<a href="/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
|
||||
Helm[<a href="https://holos.run/docs/v1alpha5/api/author/#Helm">Helm</a>]
|
||||
Kustomize[<a href="https://holos.run/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
|
||||
Kubernetes[<a href="https://holos.run/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
|
||||
|
||||
BuildPlan[<a href="/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
|
||||
BuildPlan[<a href="https://holos.run/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
|
||||
|
||||
ResourcesArtifact[<a href="/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
|
||||
GitOpsArtifact[<a href="/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
|
||||
ResourcesArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
|
||||
GitOpsArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
|
||||
|
||||
Generators[<a href="/docs/v1alpha5/api/core/#Generator">Generators</a>]
|
||||
Transformers[<a href="/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
|
||||
Validators[Validators]
|
||||
Generators[<a href="https://holos.run/docs/v1alpha5/api/core/#Generator">Generators</a>]
|
||||
Transformers[<a href="https://holos.run/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
|
||||
Validators[<a href="https://holos.run/docs/v1alpha5/api/core/#Validator">Validators</a>]
|
||||
Files[Manifest<br/>Files]
|
||||
|
||||
Platform --> Component
|
||||
|
||||
@@ -1,4 +1,16 @@
|
||||
/docs /docs/v1alpha5/ 301
|
||||
/docs/ /docs/v1alpha5/ 301
|
||||
/docs/overview /docs/v1alpha5/tutorial/overview/ 301
|
||||
/docs/overview/ /docs/v1alpha5/tutorial/overview/ 301
|
||||
/docs /docs/v1alpha5/ 301
|
||||
/docs/ /docs/v1alpha5/ 301
|
||||
/docs/tutorial /docs/v1alpha5/tutorial/ 301
|
||||
/docs/tutorial/ /docs/v1alpha5/tutorial/ 301
|
||||
/docs/quickstart /docs/v1alpha5/tutorial/overview/ 301
|
||||
/docs/quickstart/ /docs/v1alpha5/tutorial/overview/ 301
|
||||
/docs/overview /docs/v1alpha5/tutorial/overview/ 301
|
||||
/docs/overview/ /docs/v1alpha5/tutorial/overview/ 301
|
||||
/docs/topics /docs/v1alpha5/topics/structures/ 301
|
||||
/docs/topics/ /docs/v1alpha5/topics/structures/ 301
|
||||
/docs/setup /docs/v1alpha5/tutorial/setup/ 301
|
||||
/docs/setup/ /docs/v1alpha5/tutorial/setup/ 301
|
||||
/docs/local-cluster /docs/v1alpha5/topics/local-cluster/ 301
|
||||
/docs/local-cluster/ /docs/v1alpha5/topics/local-cluster/ 301
|
||||
/docs/guides/helm /docs/v1alpha5/tutorial/helm-values/ 301
|
||||
/docs/guides/helm/ /docs/v1alpha5/tutorial/helm-values/ 301
|
||||
|
||||
BIN
doc/website/static/img/cards/validators.png
Normal file
BIN
doc/website/static/img/cards/validators.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 98 KiB |
24
go.mod
24
go.mod
@@ -10,7 +10,7 @@ require (
|
||||
connectrpc.com/grpcreflect v1.2.0
|
||||
connectrpc.com/otelconnect v0.7.0
|
||||
connectrpc.com/validate v0.1.0
|
||||
cuelang.org/go v0.10.1
|
||||
cuelang.org/go v0.11.0
|
||||
entgo.io/ent v0.13.1
|
||||
github.com/bufbuild/buf v1.35.1
|
||||
github.com/choria-io/machine-room v0.0.0-20240417064836-c604da2f005e
|
||||
@@ -33,9 +33,9 @@ require (
|
||||
github.com/spf13/cobra v1.8.1
|
||||
github.com/spf13/pflag v1.0.5
|
||||
github.com/stretchr/testify v1.9.0
|
||||
golang.org/x/net v0.28.0
|
||||
golang.org/x/net v0.30.0
|
||||
golang.org/x/sync v0.8.0
|
||||
golang.org/x/tools v0.24.0
|
||||
golang.org/x/tools v0.26.0
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20240617180043-68d350f18fd4
|
||||
google.golang.org/protobuf v1.34.2
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
@@ -53,7 +53,7 @@ require (
|
||||
buf.build/gen/go/bufbuild/registry/connectrpc/go v1.16.2-20240610164129-660609bc46d3.1 // indirect
|
||||
buf.build/gen/go/bufbuild/registry/protocolbuffers/go v1.34.2-20240610164129-660609bc46d3.2 // indirect
|
||||
cloud.google.com/go/compute/metadata v0.3.0 // indirect
|
||||
cuelabs.dev/go/oci/ociregistry v0.0.0-20240807094312-a32ad29eed79 // indirect
|
||||
cuelabs.dev/go/oci/ociregistry v0.0.0-20240906074133-82eb438dd565 // indirect
|
||||
dario.cat/mergo v1.0.0 // indirect
|
||||
github.com/AlecAivazis/survey/v2 v2.3.7 // indirect
|
||||
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible // indirect
|
||||
@@ -256,7 +256,7 @@ require (
|
||||
github.com/opencontainers/image-spec v1.1.0 // indirect
|
||||
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
|
||||
github.com/pelletier/go-toml v1.9.5 // indirect
|
||||
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
|
||||
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
|
||||
github.com/pjbgf/sha1cd v0.3.0 // indirect
|
||||
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
|
||||
github.com/pkg/errors v0.9.1 // indirect
|
||||
@@ -268,7 +268,7 @@ require (
|
||||
github.com/prometheus/client_model v0.6.1 // indirect
|
||||
github.com/prometheus/common v0.52.3 // indirect
|
||||
github.com/prometheus/procfs v0.13.0 // indirect
|
||||
github.com/protocolbuffers/txtpbfmt v0.0.0-20231025115547-084445ff1adf // indirect
|
||||
github.com/protocolbuffers/txtpbfmt v0.0.0-20240823084532-8e6b51fa9bef // indirect
|
||||
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
github.com/rivo/uniseg v0.4.7 // indirect
|
||||
@@ -326,14 +326,14 @@ require (
|
||||
go.uber.org/automaxprocs v1.5.3 // indirect
|
||||
go.uber.org/multierr v1.11.0 // indirect
|
||||
go.uber.org/zap v1.27.0 // indirect
|
||||
golang.org/x/crypto v0.26.0 // indirect
|
||||
golang.org/x/crypto v0.28.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8 // indirect
|
||||
golang.org/x/exp/typeparams v0.0.0-20221208152030-732eee02a75a // indirect
|
||||
golang.org/x/mod v0.20.0 // indirect
|
||||
golang.org/x/oauth2 v0.22.0 // indirect
|
||||
golang.org/x/sys v0.23.0 // indirect
|
||||
golang.org/x/term v0.23.0 // indirect
|
||||
golang.org/x/text v0.17.0 // indirect
|
||||
golang.org/x/mod v0.21.0 // indirect
|
||||
golang.org/x/oauth2 v0.23.0 // indirect
|
||||
golang.org/x/sys v0.26.0 // indirect
|
||||
golang.org/x/term v0.25.0 // indirect
|
||||
golang.org/x/text v0.19.0 // indirect
|
||||
golang.org/x/time v0.5.0 // indirect
|
||||
google.golang.org/genproto v0.0.0-20240123012728-ef4313101c80 // indirect
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20240617180043-68d350f18fd4 // indirect
|
||||
|
||||
48
go.sum
48
go.sum
@@ -48,10 +48,10 @@ connectrpc.com/otelconnect v0.7.0 h1:ZH55ZZtcJOTKWWLy3qmL4Pam4RzRWBJFOqTPyAqCXkY
|
||||
connectrpc.com/otelconnect v0.7.0/go.mod h1:Bt2ivBymHZHqxvo4HkJ0EwHuUzQN6k2l0oH+mp/8nwc=
|
||||
connectrpc.com/validate v0.1.0 h1:r55jirxMK7HO/xZwVHj3w2XkVFarsUM77ZDy367NtH4=
|
||||
connectrpc.com/validate v0.1.0/go.mod h1:GU47c9/x/gd+u9wRSPkrQOP46gx2rMN+Wo37EHgI3Ow=
|
||||
cuelabs.dev/go/oci/ociregistry v0.0.0-20240807094312-a32ad29eed79 h1:EceZITBGET3qHneD5xowSTY/YHbNybvMWGh62K2fG/M=
|
||||
cuelabs.dev/go/oci/ociregistry v0.0.0-20240807094312-a32ad29eed79/go.mod h1:5A4xfTzHTXfeVJBU6RAUf+QrlfTCW+017q/QiW+sMLg=
|
||||
cuelang.org/go v0.10.1 h1:vDRRsd/5CICzisZ/13kBmXt3M+9eDl/pI06rrHyhlgA=
|
||||
cuelang.org/go v0.10.1/go.mod h1:HzlaqqqInHNiqE6slTP6+UtxT9hN6DAzgJgdbNxXvX8=
|
||||
cuelabs.dev/go/oci/ociregistry v0.0.0-20240906074133-82eb438dd565 h1:R5wwEcbEZSBmeyg91MJZTxfd7WpBo2jPof3AYjRbxwY=
|
||||
cuelabs.dev/go/oci/ociregistry v0.0.0-20240906074133-82eb438dd565/go.mod h1:5A4xfTzHTXfeVJBU6RAUf+QrlfTCW+017q/QiW+sMLg=
|
||||
cuelang.org/go v0.11.0 h1:2af2nhipqlUHtXk2dtOP5xnMm1ObGvKqIsJUJL1sRE4=
|
||||
cuelang.org/go v0.11.0/go.mod h1:PBY6XvPUswPPJ2inpvUozP9mebDVTXaeehQikhZPBz0=
|
||||
dario.cat/mergo v1.0.0 h1:AGCNq9Evsj31mOgNPcLyXc+4PNABt905YmuqPYYpBWk=
|
||||
dario.cat/mergo v1.0.0/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk=
|
||||
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
|
||||
@@ -734,8 +734,8 @@ github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaR
|
||||
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
|
||||
github.com/pelletier/go-toml v1.9.5 h1:4yBQzkHv+7BHq2PQUZF3Mx0IYxG7LsP222s7Agd3ve8=
|
||||
github.com/pelletier/go-toml v1.9.5/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
|
||||
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
|
||||
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
|
||||
github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M=
|
||||
github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc=
|
||||
github.com/pjbgf/sha1cd v0.3.0 h1:4D5XXmUUBUl/xQ6IjCkEAbqXskkq/4O7LmGn0AqMDs4=
|
||||
github.com/pjbgf/sha1cd v0.3.0/go.mod h1:nZ1rrWOcGJ5uZgEEVL1VUM9iRQiZvWdbZjkKyFzPPsI=
|
||||
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8/go.mod h1:HKlIX3XHQyzLZPlr7++PzdhaXEj94dEiJgZDTsxEqUI=
|
||||
@@ -769,8 +769,8 @@ github.com/prometheus/common v0.52.3 h1:5f8uj6ZwHSscOGNdIQg6OiZv/ybiK2CO2q2drVZA
|
||||
github.com/prometheus/common v0.52.3/go.mod h1:BrxBKv3FWBIGXw89Mg1AeBq7FSyRzXWI3l3e7W3RN5U=
|
||||
github.com/prometheus/procfs v0.13.0 h1:GqzLlQyfsPbaEHaQkO7tbDlriv/4o5Hudv6OXHGKX7o=
|
||||
github.com/prometheus/procfs v0.13.0/go.mod h1:cd4PFCR54QLnGKPaKGA6l+cfuNXtht43ZKY6tow0Y1g=
|
||||
github.com/protocolbuffers/txtpbfmt v0.0.0-20231025115547-084445ff1adf h1:014O62zIzQwvoD7Ekj3ePDF5bv9Xxy0w6AZk0qYbjUk=
|
||||
github.com/protocolbuffers/txtpbfmt v0.0.0-20231025115547-084445ff1adf/go.mod h1:jgxiZysxFPM+iWKwQwPR+y+Jvo54ARd4EisXxKYpB5c=
|
||||
github.com/protocolbuffers/txtpbfmt v0.0.0-20240823084532-8e6b51fa9bef h1:ej+64jiny5VETZTqcc1GFVAPEtaSk6U1D0kKC2MS5Yc=
|
||||
github.com/protocolbuffers/txtpbfmt v0.0.0-20240823084532-8e6b51fa9bef/go.mod h1:jgxiZysxFPM+iWKwQwPR+y+Jvo54ARd4EisXxKYpB5c=
|
||||
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 h1:N/ElC8H3+5XpJzTSTfLsJV/mx9Q9g7kxmchpfZyxgzM=
|
||||
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
@@ -968,8 +968,8 @@ golang.org/x/crypto v0.3.1-0.20221117191849-2c476679df9a/go.mod h1:hebNnKkNXi2Uz
|
||||
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
|
||||
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
|
||||
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
|
||||
golang.org/x/crypto v0.26.0 h1:RrRspgV4mU+YwB4FYnuBoKsUapNIL5cohGAmSH3azsw=
|
||||
golang.org/x/crypto v0.26.0/go.mod h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn54=
|
||||
golang.org/x/crypto v0.28.0 h1:GBDwsMXVQi34v5CCYUm2jkJvu4cbtru2U4TN2PSyQnw=
|
||||
golang.org/x/crypto v0.28.0/go.mod h1:rmgy+3RHxRZMyY0jjAJShp2zgEdOqj2AO7U0pYmeQ7U=
|
||||
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
|
||||
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
|
||||
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
|
||||
@@ -1007,8 +1007,8 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
|
||||
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
|
||||
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
|
||||
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
|
||||
golang.org/x/mod v0.20.0 h1:utOm6MM3R3dnawAiJgn0y+xvuYRsm1RKM/4giyfDgV0=
|
||||
golang.org/x/mod v0.20.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
|
||||
golang.org/x/mod v0.21.0 h1:vvrHzRwRfVKSiLrG+d4FMl/Qi4ukBCE6kZlTUkDYRT0=
|
||||
golang.org/x/mod v0.21.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY=
|
||||
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
@@ -1048,16 +1048,16 @@ golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
|
||||
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
|
||||
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
|
||||
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
|
||||
golang.org/x/net v0.28.0 h1:a9JDOJc5GMUJ0+UDqmLT86WiEy7iWyIhz8gz8E4e5hE=
|
||||
golang.org/x/net v0.28.0/go.mod h1:yqtgsTWOOnlGLG9GFRrK3++bGOUEkNBoHZc8MEDWPNg=
|
||||
golang.org/x/net v0.30.0 h1:AcW1SDZMkb8IpzCdQUaIq2sP4sZ4zw+55h6ynffypl4=
|
||||
golang.org/x/net v0.30.0/go.mod h1:2wGyMJ5iFasEhkwi13ChkO/t1ECNC4X4eBKkVFyYFlU=
|
||||
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20211005180243-6b3c2da341f1/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
|
||||
golang.org/x/oauth2 v0.22.0 h1:BzDx2FehcG7jJwgWLELCdmLuxk2i+x9UDpSiss2u0ZA=
|
||||
golang.org/x/oauth2 v0.22.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
|
||||
golang.org/x/oauth2 v0.23.0 h1:PbgcYx2W7i4LvjJWEbf0ngHV6qJYr86PkAV3bXdLEbs=
|
||||
golang.org/x/oauth2 v0.23.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
|
||||
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
@@ -1138,8 +1138,8 @@ golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.23.0 h1:YfKFowiIMvtgl1UERQoTPPToxltDeZfbj4H7dVUCwmM=
|
||||
golang.org/x/sys v0.23.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
|
||||
golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
|
||||
@@ -1147,8 +1147,8 @@ golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
|
||||
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
|
||||
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
|
||||
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
|
||||
golang.org/x/term v0.23.0 h1:F6D4vR+EHoL9/sWAWgAR1H2DcHr4PareCbAaCo1RpuU=
|
||||
golang.org/x/term v0.23.0/go.mod h1:DgV24QBUrK6jhZXl+20l6UWznPlwAHm1Q1mGHtydmSk=
|
||||
golang.org/x/term v0.25.0 h1:WtHI/ltw4NvSUig5KARz9h521QvRC8RmF/cuYqifU24=
|
||||
golang.org/x/term v0.25.0/go.mod h1:RPyXicDX+6vLxogjjRxjgD2TKtmAO6NZBsBRfrOLu7M=
|
||||
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
@@ -1163,8 +1163,8 @@ golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
|
||||
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
|
||||
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
|
||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc=
|
||||
golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
|
||||
golang.org/x/text v0.19.0 h1:kTxAhCbGbxhK0IwgSKiMO5awPoDQ0RpfiVYBfK860YM=
|
||||
golang.org/x/text v0.19.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
|
||||
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
@@ -1217,8 +1217,8 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f
|
||||
golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
|
||||
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
|
||||
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
|
||||
golang.org/x/tools v0.24.0 h1:J1shsA93PJUEVaUSaay7UXAyE8aimq3GW0pjlolpa24=
|
||||
golang.org/x/tools v0.24.0/go.mod h1:YhNqVBIfWHdzvTLs0d8LCuMhkKUgSUKldakyV7W/WDQ=
|
||||
golang.org/x/tools v0.26.0 h1:v/60pFQmzmT9ExmjDv2gGIfi3OqfKoEP6I5+umXlbnQ=
|
||||
golang.org/x/tools v0.26.0/go.mod h1:TPVVj70c7JJ3WCazhD8OdXcZg/og+b9+tH/KxylGwH0=
|
||||
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
|
||||
@@ -20,6 +20,7 @@ var _ Store = NewStore()
|
||||
type Store interface {
|
||||
Get(path string) (data []byte, ok bool)
|
||||
Set(path string, data []byte) error
|
||||
// Save previously set path to dir preserving directories.
|
||||
Save(dir, path string) error
|
||||
}
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
// PlatformOpts represents build options when processing the components in a
|
||||
// platform.
|
||||
type PlatformOpts struct {
|
||||
Fn BuildFunc
|
||||
Fn func(context.Context, int, holos.Component) error
|
||||
Selector holos.Selector
|
||||
Concurrency int
|
||||
InfoEnabled bool
|
||||
@@ -89,9 +89,6 @@ func (p *Platform) Build(ctx context.Context, opts PlatformOpts) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// BuildFunc is executed concurrently when processing platform components.
|
||||
type BuildFunc func(context.Context, int, holos.Component) error
|
||||
|
||||
func LoadPlatform(i *Instance) (platform Platform, err error) {
|
||||
err = i.Discriminate(func(tm holos.TypeMeta) error {
|
||||
if tm.Kind != "Platform" {
|
||||
|
||||
@@ -5,16 +5,18 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"cuelang.org/go/cue"
|
||||
core "github.com/holos-run/holos/api/core/v1alpha5"
|
||||
"github.com/holos-run/holos/internal/artifact"
|
||||
"github.com/holos-run/holos/internal/errors"
|
||||
"github.com/holos-run/holos/internal/holos"
|
||||
"github.com/holos-run/holos/internal/logger"
|
||||
@@ -107,167 +109,84 @@ func (c *Component) Path() string {
|
||||
}
|
||||
|
||||
var _ holos.BuildPlan = &BuildPlan{}
|
||||
var _ task = generatorTask{}
|
||||
var _ task = transformersTask{}
|
||||
var _ task = validatorTask{}
|
||||
|
||||
// BuildPlan represents a component builder.
|
||||
type BuildPlan struct {
|
||||
core.BuildPlan
|
||||
Opts holos.BuildOpts
|
||||
type task interface {
|
||||
id() string
|
||||
run(ctx context.Context) error
|
||||
}
|
||||
|
||||
// Build builds a BuildPlan into Artifact files.
|
||||
func (b *BuildPlan) Build(ctx context.Context) error {
|
||||
name := b.BuildPlan.Metadata.Name
|
||||
path := b.Opts.Path
|
||||
log := logger.FromContext(ctx).With("name", name, "path", path)
|
||||
msg := fmt.Sprintf("could not build %s", name)
|
||||
if b.BuildPlan.Spec.Disabled {
|
||||
log.WarnContext(ctx, fmt.Sprintf("%s: disabled", msg))
|
||||
return nil
|
||||
}
|
||||
type taskParams struct {
|
||||
taskName string
|
||||
buildPlanName string
|
||||
opts holos.BuildOpts
|
||||
}
|
||||
|
||||
g, ctx := errgroup.WithContext(ctx)
|
||||
// One more for the producer
|
||||
g.SetLimit(b.Opts.Concurrency + 1)
|
||||
func (t taskParams) id() string {
|
||||
return fmt.Sprintf("%s:%s/%s", t.opts.Path, t.buildPlanName, t.taskName)
|
||||
}
|
||||
|
||||
// Producer.
|
||||
g.Go(func() error {
|
||||
for _, a := range b.BuildPlan.Spec.Artifacts {
|
||||
msg := fmt.Sprintf("%s artifact %s", msg, a.Artifact)
|
||||
log := log.With("artifact", a.Artifact)
|
||||
if a.Skip {
|
||||
log.WarnContext(ctx, fmt.Sprintf("%s: skipped field is true", msg))
|
||||
continue
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
default:
|
||||
// https://golang.org/doc/faq#closures_and_goroutines
|
||||
a := a
|
||||
// Worker. Blocks if limit has been reached.
|
||||
g.Go(func() error {
|
||||
for _, gen := range a.Generators {
|
||||
switch gen.Kind {
|
||||
case "Resources":
|
||||
if err := b.resources(log, gen, b.Opts.Store); err != nil {
|
||||
return errors.Format("could not generate resources: %w", err)
|
||||
}
|
||||
case "Helm":
|
||||
if err := b.helm(ctx, log, gen, b.Opts.Store); err != nil {
|
||||
return errors.Format("could not generate helm: %w", err)
|
||||
}
|
||||
case "File":
|
||||
if err := b.file(log, gen, b.Opts.Store); err != nil {
|
||||
return errors.Format("could not generate file: %w", err)
|
||||
}
|
||||
default:
|
||||
return errors.Format("%s: unsupported kind %s", msg, gen.Kind)
|
||||
}
|
||||
}
|
||||
type generatorTask struct {
|
||||
taskParams
|
||||
generator core.Generator
|
||||
wg *sync.WaitGroup
|
||||
}
|
||||
|
||||
for _, t := range a.Transformers {
|
||||
switch t.Kind {
|
||||
case "Kustomize":
|
||||
if err := b.kustomize(ctx, log, t, b.Opts.Store); err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
case "Join":
|
||||
s := make([][]byte, 0, len(t.Inputs))
|
||||
for _, input := range t.Inputs {
|
||||
if data, ok := b.Opts.Store.Get(string(input)); ok {
|
||||
s = append(s, data)
|
||||
} else {
|
||||
return errors.Format("%s: missing %s", msg, input)
|
||||
}
|
||||
}
|
||||
data := bytes.Join(s, []byte(t.Join.Separator))
|
||||
if err := b.Opts.Store.Set(string(t.Output), data); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
log.Debug("set artifact: " + string(t.Output))
|
||||
default:
|
||||
return errors.Format("%s: unsupported kind %s", msg, t.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
for _, validator := range a.Validators {
|
||||
switch validator.Kind {
|
||||
case "Command":
|
||||
if err := b.validate(ctx, log, validator, b.Opts.Store); err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
default:
|
||||
return errors.Format("%s: unsupported kind %s", msg, validator.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
// Write the final artifact
|
||||
if err := b.Opts.Store.Save(b.Opts.WriteTo, string(a.Artifact)); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
log.DebugContext(ctx, "wrote "+filepath.Join(b.Opts.WriteTo, string(a.Artifact)))
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
func (t generatorTask) run(ctx context.Context) error {
|
||||
defer t.wg.Done()
|
||||
msg := fmt.Sprintf("could not build %s", t.id())
|
||||
switch t.generator.Kind {
|
||||
case "Resources":
|
||||
if err := t.resources(); err != nil {
|
||||
return errors.Format("%s: could not generate resources: %w", msg, err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
// Wait for completion and return the first error (if any)
|
||||
return g.Wait()
|
||||
}
|
||||
|
||||
func (b *BuildPlan) Export(idx int, encoder holos.OrderedEncoder) error {
|
||||
if err := encoder.Encode(idx, &b.BuildPlan); err != nil {
|
||||
return errors.Wrap(err)
|
||||
case "Helm":
|
||||
if err := t.helm(ctx); err != nil {
|
||||
return errors.Format("%s: could not generate helm: %w", msg, err)
|
||||
}
|
||||
case "File":
|
||||
if err := t.file(); err != nil {
|
||||
return errors.Format("%s: could not generate file: %w", msg, err)
|
||||
}
|
||||
default:
|
||||
return errors.Format("%s: unsupported kind %s", msg, t.generator.Kind)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *BuildPlan) Load(v cue.Value) error {
|
||||
return errors.Wrap(v.Decode(&b.BuildPlan))
|
||||
}
|
||||
|
||||
func (b *BuildPlan) file(
|
||||
log *slog.Logger,
|
||||
g core.Generator,
|
||||
store artifact.Store,
|
||||
) error {
|
||||
data, err := os.ReadFile(filepath.Join(string(b.Opts.Path), string(g.File.Source)))
|
||||
func (t generatorTask) file() error {
|
||||
data, err := os.ReadFile(filepath.Join(string(t.opts.Path), string(t.generator.File.Source)))
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
if err := store.Set(string(g.Output), data); err != nil {
|
||||
if err := t.opts.Store.Set(string(t.generator.Output), data); err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
log.Debug("set artifact: " + string(g.Output))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *BuildPlan) helm(
|
||||
ctx context.Context,
|
||||
log *slog.Logger,
|
||||
g core.Generator,
|
||||
store artifact.Store,
|
||||
) error {
|
||||
chartName := g.Helm.Chart.Name
|
||||
log = log.With("chart", chartName)
|
||||
func (t generatorTask) helm(ctx context.Context) error {
|
||||
chartName := t.generator.Helm.Chart.Name
|
||||
// Unnecessary? cargo cult copied from internal/cli/render/render.go
|
||||
if chartName == "" {
|
||||
return errors.New("missing chart name")
|
||||
}
|
||||
|
||||
// Cache the chart by version to pull new versions. (#273)
|
||||
cacheDir := filepath.Join(string(b.Opts.Path), "vendor", g.Helm.Chart.Version)
|
||||
cacheDir := filepath.Join(string(t.opts.Path), "vendor", t.generator.Helm.Chart.Version)
|
||||
cachePath := filepath.Join(cacheDir, filepath.Base(chartName))
|
||||
|
||||
log := logger.FromContext(ctx)
|
||||
|
||||
if _, err := os.Stat(cachePath); os.IsNotExist(err) {
|
||||
timeout, cancel := context.WithTimeout(ctx, 5*time.Minute)
|
||||
defer cancel()
|
||||
err := onceWithLock(log, timeout, cachePath, func() error {
|
||||
return b.cacheChart(ctx, log, cacheDir, g.Helm.Chart)
|
||||
})
|
||||
err := func() error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
|
||||
defer cancel()
|
||||
return onceWithLock(log, ctx, cachePath, func() error {
|
||||
return cacheChart(ctx, cacheDir, t.generator.Helm.Chart, t.opts.Stderr)
|
||||
})
|
||||
}()
|
||||
if err != nil {
|
||||
return errors.Format("could not cache chart: %w", err)
|
||||
}
|
||||
@@ -280,7 +199,7 @@ func (b *BuildPlan) helm(
|
||||
}
|
||||
defer util.Remove(ctx, tempDir)
|
||||
|
||||
data, err := yaml.Marshal(g.Helm.Values)
|
||||
data, err := yaml.Marshal(t.generator.Helm.Values)
|
||||
if err != nil {
|
||||
return errors.Format("could not marshal values: %w", err)
|
||||
}
|
||||
@@ -293,22 +212,22 @@ func (b *BuildPlan) helm(
|
||||
|
||||
// Run charts
|
||||
args := []string{"template"}
|
||||
if !g.Helm.EnableHooks {
|
||||
if !t.generator.Helm.EnableHooks {
|
||||
args = append(args, "--no-hooks")
|
||||
}
|
||||
for _, apiVersion := range g.Helm.APIVersions {
|
||||
for _, apiVersion := range t.generator.Helm.APIVersions {
|
||||
args = append(args, "--api-versions", apiVersion)
|
||||
}
|
||||
if kubeVersion := g.Helm.KubeVersion; kubeVersion != "" {
|
||||
if kubeVersion := t.generator.Helm.KubeVersion; kubeVersion != "" {
|
||||
args = append(args, "--kube-version", kubeVersion)
|
||||
}
|
||||
args = append(args,
|
||||
"--include-crds",
|
||||
"--values", valuesPath,
|
||||
"--namespace", g.Helm.Namespace,
|
||||
"--namespace", t.generator.Helm.Namespace,
|
||||
"--kubeconfig", "/dev/null",
|
||||
"--version", g.Helm.Chart.Version,
|
||||
g.Helm.Chart.Release,
|
||||
"--version", t.generator.Helm.Chart.Version,
|
||||
t.generator.Helm.Chart.Release,
|
||||
cachePath,
|
||||
)
|
||||
helmOut, err := util.RunCmd(ctx, "helm", args...)
|
||||
@@ -325,36 +244,31 @@ func (b *BuildPlan) helm(
|
||||
}
|
||||
|
||||
// Set the artifact
|
||||
if err := store.Set(string(g.Output), helmOut.Stdout.Bytes()); err != nil {
|
||||
if err := t.opts.Store.Set(string(t.generator.Output), helmOut.Stdout.Bytes()); err != nil {
|
||||
return errors.Format("could not store helm output: %w", err)
|
||||
}
|
||||
log.Debug("set artifact: " + string(g.Output))
|
||||
log.Debug("set artifact: " + string(t.generator.Output))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *BuildPlan) resources(
|
||||
log *slog.Logger,
|
||||
g core.Generator,
|
||||
store artifact.Store,
|
||||
) error {
|
||||
func (t generatorTask) resources() error {
|
||||
var size int
|
||||
for _, m := range g.Resources {
|
||||
for _, m := range t.generator.Resources {
|
||||
size += len(m)
|
||||
}
|
||||
list := make([]core.Resource, 0, size)
|
||||
|
||||
for _, m := range g.Resources {
|
||||
for _, m := range t.generator.Resources {
|
||||
for _, r := range m {
|
||||
list = append(list, r)
|
||||
}
|
||||
}
|
||||
|
||||
msg := fmt.Sprintf(
|
||||
"could not generate %s for %s path %s",
|
||||
g.Output,
|
||||
b.BuildPlan.Metadata.Name,
|
||||
b.Opts.Path,
|
||||
"could not generate %s for %s",
|
||||
t.generator.Output,
|
||||
t.id(),
|
||||
)
|
||||
|
||||
buf, err := marshal(list)
|
||||
@@ -362,114 +276,220 @@ func (b *BuildPlan) resources(
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
|
||||
if err := store.Set(string(g.Output), buf.Bytes()); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
|
||||
log.Debug("set artifact " + string(g.Output))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *BuildPlan) validate(
|
||||
ctx context.Context,
|
||||
log *slog.Logger,
|
||||
validator core.Validator,
|
||||
store artifact.Store,
|
||||
) error {
|
||||
tempDir, err := os.MkdirTemp("", "holos.validate")
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
defer util.Remove(ctx, tempDir)
|
||||
msg := fmt.Sprintf(
|
||||
"could not validate %s path %s",
|
||||
b.BuildPlan.Metadata.Name,
|
||||
b.Opts.Path,
|
||||
)
|
||||
|
||||
// Write the inputs
|
||||
for _, input := range validator.Inputs {
|
||||
path := string(input)
|
||||
if err := store.Save(tempDir, path); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
log.DebugContext(ctx, "wrote "+filepath.Join(tempDir, path))
|
||||
}
|
||||
|
||||
if len(validator.Command.Args) < 1 {
|
||||
return errors.Format("%s: command args length must be at least 1", msg)
|
||||
}
|
||||
size := len(validator.Command.Args) + len(validator.Inputs)
|
||||
args := make([]string, 0, size)
|
||||
args = append(args, validator.Command.Args...)
|
||||
for _, input := range validator.Inputs {
|
||||
args = append(args, filepath.Join(tempDir, string(input)))
|
||||
}
|
||||
|
||||
// Execute the validator
|
||||
if _, err = util.RunCmdA(ctx, b.Opts.Stderr, args[0], args[1:]...); err != nil {
|
||||
if err := t.opts.Store.Set(string(t.generator.Output), buf.Bytes()); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *BuildPlan) kustomize(
|
||||
ctx context.Context,
|
||||
log *slog.Logger,
|
||||
t core.Transformer,
|
||||
store artifact.Store,
|
||||
) error {
|
||||
tempDir, err := os.MkdirTemp("", "holos.kustomize")
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
defer util.Remove(ctx, tempDir)
|
||||
msg := fmt.Sprintf(
|
||||
"could not transform %s for %s path %s",
|
||||
t.Output,
|
||||
b.BuildPlan.Metadata.Name,
|
||||
b.Opts.Path,
|
||||
)
|
||||
type transformersTask struct {
|
||||
taskParams
|
||||
transformers []core.Transformer
|
||||
wg *sync.WaitGroup
|
||||
}
|
||||
|
||||
// Write the kustomization
|
||||
data, err := yaml.Marshal(t.Kustomize.Kustomization)
|
||||
if err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
path := filepath.Join(tempDir, "kustomization.yaml")
|
||||
if err := os.WriteFile(path, data, 0666); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
log.DebugContext(ctx, "wrote "+path)
|
||||
|
||||
// Write the inputs
|
||||
for _, input := range t.Inputs {
|
||||
path := string(input)
|
||||
if err := store.Save(tempDir, path); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
func (t transformersTask) run(ctx context.Context) error {
|
||||
defer t.wg.Done()
|
||||
for idx, transformer := range t.transformers {
|
||||
msg := fmt.Sprintf("could not build %s/%d", t.id(), idx)
|
||||
switch transformer.Kind {
|
||||
case "Kustomize":
|
||||
if err := kustomize(ctx, transformer, t.taskParams); err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
case "Join":
|
||||
s := make([][]byte, 0, len(transformer.Inputs))
|
||||
for _, input := range transformer.Inputs {
|
||||
if data, ok := t.opts.Store.Get(string(input)); ok {
|
||||
s = append(s, data)
|
||||
} else {
|
||||
return errors.Format("%s: missing %s", msg, input)
|
||||
}
|
||||
}
|
||||
data := bytes.Join(s, []byte(transformer.Join.Separator))
|
||||
if err := t.opts.Store.Set(string(transformer.Output), data); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
default:
|
||||
return errors.Format("%s: unsupported kind %s", msg, transformer.Kind)
|
||||
}
|
||||
log.DebugContext(ctx, "wrote "+filepath.Join(tempDir, path))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Execute kustomize
|
||||
r, err := util.RunCmdW(ctx, b.Opts.Stderr, "kubectl", "kustomize", tempDir)
|
||||
if err != nil {
|
||||
kErr := r.Stderr.String()
|
||||
err = errors.Format("%s: could not run kustomize: %w", msg, err)
|
||||
log.ErrorContext(ctx, fmt.Sprintf("%s: stderr:\n%s", err.Error(), kErr), "err", err, "stderr", kErr)
|
||||
return err
|
||||
type validatorTask struct {
|
||||
taskParams
|
||||
validator core.Validator
|
||||
wg *sync.WaitGroup
|
||||
}
|
||||
|
||||
func (t validatorTask) run(ctx context.Context) error {
|
||||
defer t.wg.Done()
|
||||
msg := fmt.Sprintf("could not validate %s", t.id())
|
||||
switch kind := t.validator.Kind; kind {
|
||||
case "Command":
|
||||
if err := validate(ctx, t.validator, t.taskParams); err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
default:
|
||||
return errors.Format("%s: unsupported kind %s", msg, kind)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Store the artifact
|
||||
if err := store.Set(string(t.Output), r.Stdout.Bytes()); err != nil {
|
||||
func worker(ctx context.Context, idx int, tasks chan task) error {
|
||||
log := logger.FromContext(ctx).With("worker", idx)
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case task, ok := <-tasks:
|
||||
if !ok {
|
||||
log.DebugContext(ctx, fmt.Sprintf("worker %d returning: tasks chan closed", idx))
|
||||
return nil
|
||||
}
|
||||
log.DebugContext(ctx, fmt.Sprintf("worker %d task %s starting", idx, task.id()))
|
||||
if err := task.run(ctx); err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
log.DebugContext(ctx, fmt.Sprintf("worker %d task %s finished ok", idx, task.id()))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func buildArtifact(ctx context.Context, idx int, artifact core.Artifact, tasks chan task, buildPlanName string, opts holos.BuildOpts) error {
|
||||
var wg sync.WaitGroup
|
||||
msg := fmt.Sprintf("could not build %s artifact %s", buildPlanName, artifact.Artifact)
|
||||
// Process Generators concurrently
|
||||
for gid, gen := range artifact.Generators {
|
||||
task := generatorTask{
|
||||
taskParams: taskParams{
|
||||
taskName: fmt.Sprintf("artifact/%d/generator/%d", idx, gid),
|
||||
buildPlanName: buildPlanName,
|
||||
opts: opts,
|
||||
},
|
||||
generator: gen,
|
||||
wg: &wg,
|
||||
}
|
||||
wg.Add(1)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case tasks <- task:
|
||||
}
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
// Process Transformers sequentially
|
||||
task := transformersTask{
|
||||
taskParams: taskParams{
|
||||
taskName: fmt.Sprintf("artifact/%d/transformers", idx),
|
||||
buildPlanName: buildPlanName,
|
||||
opts: opts,
|
||||
},
|
||||
transformers: artifact.Transformers,
|
||||
wg: &wg,
|
||||
}
|
||||
wg.Add(1)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case tasks <- task:
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
// Process Validators concurrently
|
||||
for vid, val := range artifact.Validators {
|
||||
task := validatorTask{
|
||||
taskParams: taskParams{
|
||||
taskName: fmt.Sprintf("artifact/%d/validator/%d", idx, vid),
|
||||
buildPlanName: buildPlanName,
|
||||
opts: opts,
|
||||
},
|
||||
validator: val,
|
||||
wg: &wg,
|
||||
}
|
||||
wg.Add(1)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case tasks <- task:
|
||||
}
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
// Write the final artifact
|
||||
out := string(artifact.Artifact)
|
||||
if err := opts.Store.Save(opts.WriteTo, out); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
log.Debug("set artifact " + string(t.Output))
|
||||
log := logger.FromContext(ctx)
|
||||
log.DebugContext(ctx, fmt.Sprintf("wrote %s", filepath.Join(opts.WriteTo, out)))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// BuildPlan represents a component builder.
|
||||
type BuildPlan struct {
|
||||
core.BuildPlan
|
||||
Opts holos.BuildOpts
|
||||
}
|
||||
|
||||
func (b *BuildPlan) Build(ctx context.Context) error {
|
||||
name := b.BuildPlan.Metadata.Name
|
||||
path := b.Opts.Path
|
||||
log := logger.FromContext(ctx).With("name", name, "path", path)
|
||||
|
||||
msg := fmt.Sprintf("could not build %s", name)
|
||||
if b.BuildPlan.Spec.Disabled {
|
||||
log.WarnContext(ctx, fmt.Sprintf("%s: disabled", msg))
|
||||
return nil
|
||||
}
|
||||
|
||||
g, ctx := errgroup.WithContext(ctx)
|
||||
tasks := make(chan task)
|
||||
|
||||
// Start the worker pool.
|
||||
for idx := 0; idx < max(1, b.Opts.Concurrency); idx++ {
|
||||
g.Go(func() error {
|
||||
return worker(ctx, idx, tasks)
|
||||
})
|
||||
}
|
||||
|
||||
// Start one producer that fans out to one pipeline per artifact.
|
||||
g.Go(func() error {
|
||||
// Close the tasks chan when the producer returns.
|
||||
defer func() {
|
||||
log.DebugContext(ctx, "producer returning: closing tasks chan")
|
||||
close(tasks)
|
||||
}()
|
||||
// Separate error group for producers.
|
||||
p, ctx := errgroup.WithContext(ctx)
|
||||
for idx, a := range b.BuildPlan.Spec.Artifacts {
|
||||
p.Go(func() error {
|
||||
return buildArtifact(ctx, idx, a, tasks, b.Metadata.Name, b.Opts)
|
||||
})
|
||||
}
|
||||
// Wait on producers to finish.
|
||||
return errors.Wrap(p.Wait())
|
||||
})
|
||||
|
||||
// Wait on workers to finish.
|
||||
return g.Wait()
|
||||
}
|
||||
|
||||
func (b *BuildPlan) Export(idx int, encoder holos.OrderedEncoder) error {
|
||||
if err := encoder.Encode(idx, &b.BuildPlan); err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *BuildPlan) Load(v cue.Value) error {
|
||||
return errors.Wrap(v.Decode(&b.BuildPlan))
|
||||
}
|
||||
|
||||
func marshal(list []core.Resource) (buf bytes.Buffer, err error) {
|
||||
encoder := yaml.NewEncoder(&buf)
|
||||
defer encoder.Close()
|
||||
@@ -482,95 +502,6 @@ func marshal(list []core.Resource) (buf bytes.Buffer, err error) {
|
||||
return
|
||||
}
|
||||
|
||||
// cacheChart stores a cached copy of Chart in the chart subdirectory of path.
|
||||
//
|
||||
// We assume the only method responsible for writing to chartDir is cacheChart
|
||||
// itself. cacheChart runs concurrently when rendering a platform.
|
||||
//
|
||||
// We rely on the atomicity of moving temporary directories into place on the
|
||||
// same filesystem via os.Rename. If a syscall.EEXIST error occurs during
|
||||
// renaming, it indicates that the cached chart already exists, which is
|
||||
// expected when this function is called concurrently.
|
||||
//
|
||||
// TODO(jeff): Break the dependency on v1alpha5, make it work across versions as
|
||||
// a utility function.
|
||||
func (b *BuildPlan) cacheChart(
|
||||
ctx context.Context,
|
||||
log *slog.Logger,
|
||||
cacheDir string,
|
||||
chart core.Chart,
|
||||
) error {
|
||||
// Add repositories
|
||||
repo := chart.Repository
|
||||
stderr := b.Opts.Stderr
|
||||
if repo.URL == "" {
|
||||
// repo update not needed for oci charts so this is debug instead of warn.
|
||||
log.DebugContext(ctx, "skipped helm repo add and update: repo url is empty")
|
||||
} else {
|
||||
if _, err := util.RunCmdW(ctx, stderr, "helm", "repo", "add", repo.Name, repo.URL); err != nil {
|
||||
return errors.Format("could not run helm repo add: %w", err)
|
||||
}
|
||||
if _, err := util.RunCmdW(ctx, stderr, "helm", "repo", "update", repo.Name); err != nil {
|
||||
return errors.Format("could not run helm repo update: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
cacheTemp, err := os.MkdirTemp(cacheDir, chart.Name)
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
defer util.Remove(ctx, cacheTemp)
|
||||
|
||||
cn := chart.Name
|
||||
if chart.Repository.Name != "" {
|
||||
cn = fmt.Sprintf("%s/%s", chart.Repository.Name, chart.Name)
|
||||
}
|
||||
helmOut, err := util.RunCmdW(ctx, stderr, "helm", "pull", "--destination", cacheTemp, "--untar=true", "--version", chart.Version, cn)
|
||||
if err != nil {
|
||||
stderr := helmOut.Stderr.String()
|
||||
lines := strings.Split(stderr, "\n")
|
||||
for _, line := range lines {
|
||||
log.DebugContext(ctx, line)
|
||||
if strings.HasPrefix(line, "Error:") {
|
||||
err = fmt.Errorf("%s: %w", line, err)
|
||||
}
|
||||
}
|
||||
return errors.Format("could not run helm pull: %w", err)
|
||||
}
|
||||
log.Debug("helm pull", "stdout", helmOut.Stdout, "stderr", helmOut.Stderr)
|
||||
|
||||
items, err := os.ReadDir(cacheTemp)
|
||||
if err != nil {
|
||||
return errors.Wrap(fmt.Errorf("could not read directory: %w", err))
|
||||
}
|
||||
if len(items) != 1 {
|
||||
return errors.Format("want: exactly one item, have: %+v", items)
|
||||
}
|
||||
item := items[0]
|
||||
|
||||
src := filepath.Join(cacheTemp, item.Name())
|
||||
dst := filepath.Join(cacheDir, chart.Name)
|
||||
if err := os.Rename(src, dst); err != nil {
|
||||
var linkErr *os.LinkError
|
||||
if errors.As(err, &linkErr) && errors.Is(linkErr.Err, syscall.EEXIST) {
|
||||
log.DebugContext(ctx, "cache already exists", "chart", chart.Name, "chart_version", chart.Version, "path", dst)
|
||||
} else {
|
||||
return errors.Wrap(fmt.Errorf("could not rename: %w", err))
|
||||
}
|
||||
} else {
|
||||
log.DebugContext(ctx, fmt.Sprintf("renamed %s to %s", src, dst), "src", src, "dst", dst)
|
||||
}
|
||||
|
||||
log.InfoContext(ctx,
|
||||
fmt.Sprintf("cached %s %s", chart.Name, chart.Version),
|
||||
"chart", chart.Name,
|
||||
"chart_version", chart.Version,
|
||||
"path", dst,
|
||||
)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// onceWithLock obtains a filesystem lock with mkdir, then executes fn. If the
|
||||
// lock is already locked, onceWithLock waits for it to be released then returns
|
||||
// without calling fn.
|
||||
@@ -619,3 +550,158 @@ func onceWithLock(log *slog.Logger, ctx context.Context, path string, fn func()
|
||||
// Unexpected error
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
|
||||
func cacheChart(ctx context.Context, cacheDir string, chart core.Chart, stderr io.Writer) error {
|
||||
log := logger.FromContext(ctx)
|
||||
// Add repositories
|
||||
repo := chart.Repository
|
||||
if repo.URL == "" {
|
||||
// repo update not needed for oci charts so this is debug instead of warn.
|
||||
log.DebugContext(ctx, "skipped helm repo add and update: repo url is empty")
|
||||
} else {
|
||||
if _, err := util.RunCmdW(ctx, stderr, "helm", "repo", "add", repo.Name, repo.URL); err != nil {
|
||||
return errors.Format("could not run helm repo add: %w", err)
|
||||
}
|
||||
if _, err := util.RunCmdW(ctx, stderr, "helm", "repo", "update", repo.Name); err != nil {
|
||||
return errors.Format("could not run helm repo update: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Support chart.Name = "oci:/ghcr.io/akuity/kargo-charts/kargo"
|
||||
chartBaseName := path.Base(chart.Name)
|
||||
|
||||
cacheTemp, err := os.MkdirTemp(cacheDir, chartBaseName)
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
defer util.Remove(ctx, cacheTemp)
|
||||
|
||||
cn := chart.Name
|
||||
if chart.Repository.Name != "" {
|
||||
cn = fmt.Sprintf("%s/%s", chart.Repository.Name, chart.Name)
|
||||
}
|
||||
helmOut, err := util.RunCmdW(ctx, stderr, "helm", "pull", "--destination", cacheTemp, "--untar=true", "--version", chart.Version, cn)
|
||||
if err != nil {
|
||||
stderr := helmOut.Stderr.String()
|
||||
lines := strings.Split(stderr, "\n")
|
||||
for _, line := range lines {
|
||||
log.DebugContext(ctx, line)
|
||||
if strings.HasPrefix(line, "Error:") {
|
||||
err = fmt.Errorf("%s: %w", line, err)
|
||||
}
|
||||
}
|
||||
return errors.Format("could not run helm pull: %w", err)
|
||||
}
|
||||
log.Debug("helm pull", "stdout", helmOut.Stdout, "stderr", helmOut.Stderr)
|
||||
|
||||
items, err := os.ReadDir(cacheTemp)
|
||||
if err != nil {
|
||||
return errors.Wrap(fmt.Errorf("could not read directory: %w", err))
|
||||
}
|
||||
if len(items) != 1 {
|
||||
return errors.Format("want: exactly one item, have: %+v", items)
|
||||
}
|
||||
item := items[0]
|
||||
|
||||
src := filepath.Join(cacheTemp, item.Name())
|
||||
dst := filepath.Join(cacheDir, chartBaseName)
|
||||
if err := os.Rename(src, dst); err != nil {
|
||||
var linkErr *os.LinkError
|
||||
if errors.As(err, &linkErr) && errors.Is(linkErr.Err, syscall.EEXIST) {
|
||||
log.DebugContext(ctx, "cache already exists", "chart", chart.Name, "chart_version", chart.Version, "path", dst)
|
||||
} else {
|
||||
return errors.Wrap(fmt.Errorf("could not rename: %w", err))
|
||||
}
|
||||
} else {
|
||||
log.DebugContext(ctx, fmt.Sprintf("renamed %s to %s", src, dst), "src", src, "dst", dst)
|
||||
}
|
||||
|
||||
log.InfoContext(ctx,
|
||||
fmt.Sprintf("cached %s %s", chart.Name, chart.Version),
|
||||
"chart", chart.Name,
|
||||
"chart_version", chart.Version,
|
||||
"path", dst,
|
||||
)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func kustomize(ctx context.Context, t core.Transformer, p taskParams) error {
|
||||
tempDir, err := os.MkdirTemp("", "holos.kustomize")
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
defer util.Remove(ctx, tempDir)
|
||||
msg := fmt.Sprintf(
|
||||
"could not transform %s for %s path %s",
|
||||
t.Output,
|
||||
p.buildPlanName,
|
||||
p.opts.Path,
|
||||
)
|
||||
|
||||
// Write the kustomization
|
||||
data, err := yaml.Marshal(t.Kustomize.Kustomization)
|
||||
if err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
path := filepath.Join(tempDir, "kustomization.yaml")
|
||||
if err := os.WriteFile(path, data, 0666); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
|
||||
// Write the inputs
|
||||
for _, input := range t.Inputs {
|
||||
path := string(input)
|
||||
if err := p.opts.Store.Save(tempDir, path); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Execute kustomize
|
||||
r, err := util.RunCmdW(ctx, p.opts.Stderr, "kubectl", "kustomize", tempDir)
|
||||
if err != nil {
|
||||
return errors.Format("%s: could not run kustomize: %w", msg, err)
|
||||
}
|
||||
|
||||
// Store the artifact
|
||||
if err := p.opts.Store.Set(string(t.Output), r.Stdout.Bytes()); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func validate(ctx context.Context, validator core.Validator, p taskParams) error {
|
||||
store := p.opts.Store
|
||||
tempDir, err := os.MkdirTemp("", "holos.validate")
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
// defer util.Remove(ctx, tempDir)
|
||||
msg := fmt.Sprintf("could not validate %s", p.id())
|
||||
|
||||
// Write the inputs
|
||||
for _, input := range validator.Inputs {
|
||||
path := string(input)
|
||||
if err := store.Save(tempDir, path); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
}
|
||||
|
||||
if len(validator.Command.Args) < 1 {
|
||||
return errors.Format("%s: command args length must be at least 1", msg)
|
||||
}
|
||||
size := len(validator.Command.Args) + len(validator.Inputs)
|
||||
args := make([]string, 0, size)
|
||||
args = append(args, validator.Command.Args...)
|
||||
for _, input := range validator.Inputs {
|
||||
args = append(args, filepath.Join(tempDir, string(input)))
|
||||
}
|
||||
|
||||
// Execute the validator
|
||||
if _, err = util.RunCmdA(ctx, p.opts.Stderr, args[0], args[1:]...); err != nil {
|
||||
return errors.Format("%s: %w", msg, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -4,6 +4,9 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"runtime/pprof"
|
||||
"runtime/trace"
|
||||
|
||||
"connectrpc.com/connect"
|
||||
cue "cuelang.org/go/cue/errors"
|
||||
@@ -12,12 +15,48 @@ import (
|
||||
"google.golang.org/genproto/googleapis/rpc/errdetails"
|
||||
)
|
||||
|
||||
func memProfile(ctx context.Context, cfg *holos.Config) {
|
||||
if format := os.Getenv("HOLOS_MEM_PROFILE"); format != "" {
|
||||
f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
|
||||
defer f.Close()
|
||||
if err := pprof.WriteHeapProfile(f); err != nil {
|
||||
_ = HandleError(ctx, err, cfg)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// MakeMain makes a main function for the cli or tests.
|
||||
func MakeMain(options ...holos.Option) func() int {
|
||||
return func() (exitCode int) {
|
||||
cfg := holos.New(options...)
|
||||
slog.SetDefault(cfg.Logger())
|
||||
ctx := context.Background()
|
||||
|
||||
if format := os.Getenv("HOLOS_CPU_PROFILE"); format != "" {
|
||||
f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
|
||||
err := pprof.StartCPUProfile(f)
|
||||
defer func() {
|
||||
pprof.StopCPUProfile()
|
||||
f.Close()
|
||||
}()
|
||||
if err != nil {
|
||||
return HandleError(ctx, err, cfg)
|
||||
}
|
||||
}
|
||||
defer memProfile(ctx, cfg)
|
||||
|
||||
if format := os.Getenv("HOLOS_TRACE"); format != "" {
|
||||
f, _ := os.Create(fmt.Sprintf(format, os.Getppid(), os.Getpid()))
|
||||
err := trace.Start(f)
|
||||
defer func() {
|
||||
trace.Stop()
|
||||
f.Close()
|
||||
}()
|
||||
if err != nil {
|
||||
return HandleError(ctx, err, cfg)
|
||||
}
|
||||
}
|
||||
|
||||
feature := &holos.EnvFlagger{}
|
||||
if err := New(cfg, feature).ExecuteContext(ctx); err != nil {
|
||||
return HandleError(ctx, err, cfg)
|
||||
|
||||
@@ -16,6 +16,8 @@ import (
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
const tagHelp = "set the value of a cue @tag field in the form key [ = value ]"
|
||||
|
||||
func New(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
|
||||
cmd := command.New("render")
|
||||
cmd.Args = cobra.NoArgs
|
||||
@@ -26,7 +28,7 @@ func New(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
|
||||
}
|
||||
|
||||
func newPlatform(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
|
||||
cmd := command.New("platform DIRECTORY")
|
||||
cmd := command.New("platform")
|
||||
cmd.Args = cobra.MaximumNArgs(1)
|
||||
cmd.Example = "holos render platform"
|
||||
cmd.Short = "render an entire platform"
|
||||
@@ -38,13 +40,13 @@ func newPlatform(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
|
||||
}
|
||||
|
||||
var concurrency int
|
||||
cmd.Flags().IntVar(&concurrency, "concurrency", min(runtime.NumCPU(), 8), "number of components to render concurrently")
|
||||
cmd.Flags().IntVar(&concurrency, "concurrency", runtime.NumCPU(), "number of components to render concurrently")
|
||||
var platform string
|
||||
cmd.Flags().StringVar(&platform, "platform", "./platform", "platform directory path")
|
||||
var selector holos.Selector
|
||||
cmd.Flags().VarP(&selector, "selector", "l", "label selector (e.g. label==string,label!=string)")
|
||||
tagMap := make(holos.TagMap)
|
||||
cmd.Flags().VarP(&tagMap, "inject", "t", "set the value of a cue @tag field from a key=value pair")
|
||||
cmd.Flags().VarP(&tagMap, "inject", "t", tagHelp)
|
||||
|
||||
cmd.RunE = func(cmd *cobra.Command, args []string) error {
|
||||
ctx := cmd.Root().Context()
|
||||
@@ -70,7 +72,7 @@ func newPlatform(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
|
||||
"--log-format", cfg.LogConfig().Format(),
|
||||
}
|
||||
opts := builder.PlatformOpts{
|
||||
Fn: makePlatformRenderFunc(cmd.ErrOrStderr(), prefixArgs),
|
||||
Fn: makeComponentRenderFunc(cmd.ErrOrStderr(), prefixArgs, tagMap.Tags()),
|
||||
Selector: selector,
|
||||
Concurrency: concurrency,
|
||||
InfoEnabled: true,
|
||||
@@ -102,9 +104,9 @@ func newComponent(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
|
||||
}
|
||||
|
||||
tagMap := make(holos.TagMap)
|
||||
cmd.Flags().VarP(&tagMap, "inject", "t", "set the value of a cue @tag field from a key=value pair")
|
||||
cmd.Flags().VarP(&tagMap, "inject", "t", tagHelp)
|
||||
var concurrency int
|
||||
cmd.Flags().IntVar(&concurrency, "concurrency", min(runtime.NumCPU(), 8), "number of concurrent build steps")
|
||||
cmd.Flags().IntVar(&concurrency, "concurrency", runtime.NumCPU(), "number of concurrent build steps")
|
||||
|
||||
cmd.RunE = func(cmd *cobra.Command, args []string) error {
|
||||
ctx := cmd.Root().Context()
|
||||
@@ -134,7 +136,7 @@ func newComponent(cfg *holos.Config, feature holos.Flagger) *cobra.Command {
|
||||
return cmd
|
||||
}
|
||||
|
||||
func makePlatformRenderFunc(w io.Writer, prefixArgs []string) builder.BuildFunc {
|
||||
func makeComponentRenderFunc(w io.Writer, prefixArgs, cliTags []string) func(context.Context, int, holos.Component) error {
|
||||
return func(ctx context.Context, idx int, component holos.Component) error {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
@@ -147,6 +149,9 @@ func makePlatformRenderFunc(w io.Writer, prefixArgs []string) builder.BuildFunc
|
||||
args := make([]string, 0, 10+len(prefixArgs)+(len(tags)*2))
|
||||
args = append(args, prefixArgs...)
|
||||
args = append(args, "render", "component")
|
||||
for _, tag := range cliTags {
|
||||
args = append(args, "--inject", tag)
|
||||
}
|
||||
for _, tag := range tags {
|
||||
args = append(args, "--inject", tag)
|
||||
}
|
||||
|
||||
@@ -94,6 +94,7 @@ func newShowBuildPlanCmd() (cmd *cobra.Command) {
|
||||
buildPlanOpts := holos.NewBuildOpts(path)
|
||||
buildPlanOpts.Stderr = cmd.ErrOrStderr()
|
||||
buildPlanOpts.Concurrency = concurrency
|
||||
buildPlanOpts.Tags = tagMap.Tags()
|
||||
|
||||
platformOpts := builder.PlatformOpts{
|
||||
Fn: makeBuildFunc(encoder, buildPlanOpts),
|
||||
@@ -110,7 +111,7 @@ func newShowBuildPlanCmd() (cmd *cobra.Command) {
|
||||
return cmd
|
||||
}
|
||||
|
||||
func makeBuildFunc(encoder holos.OrderedEncoder, opts holos.BuildOpts) builder.BuildFunc {
|
||||
func makeBuildFunc(encoder holos.OrderedEncoder, opts holos.BuildOpts) func(context.Context, int, holos.Component) error {
|
||||
return func(ctx context.Context, idx int, component holos.Component) error {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
@@ -120,6 +121,7 @@ func makeBuildFunc(encoder holos.OrderedEncoder, opts holos.BuildOpts) builder.B
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
}
|
||||
tags = append(tags, opts.Tags...)
|
||||
inst, err := builder.LoadInstance(component.Path(), tags)
|
||||
if err != nil {
|
||||
return errors.Wrap(err)
|
||||
|
||||
@@ -232,7 +232,10 @@ package core
|
||||
#FileContent: string
|
||||
|
||||
// Validator validates files. Useful to validate an [Artifact] prior to writing
|
||||
// it out to the final destination. Validators may be executed concurrently.
|
||||
// it out to the final destination. Holos may execute validators concurrently.
|
||||
// See the [validators] tutorial for an end to end example.
|
||||
//
|
||||
// [validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
|
||||
#Validator: {
|
||||
// Kind represents the kind of transformer. Must be Kustomize, or Join.
|
||||
kind: string & "Command" @go(Kind)
|
||||
|
||||
@@ -40,13 +40,22 @@ func (i *StringSlice) Set(value string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// TagMap represents a map of key values for CUE TagMap for flag parsing.
|
||||
type TagMap map[string]string
|
||||
// TagMap represents a map of key values for CUE TagMap for flag parsing. The
|
||||
// values are pointers to disambiguate between the case where a tag is a boolean
|
||||
// ("--inject foo") and the case where a tag has a string zero value ("--inject
|
||||
// foo="). Refer to the Tags field of [cue/load.Config]
|
||||
//
|
||||
// [cue/load.Config]: https://pkg.go.dev/cuelang.org/go@v0.10.1/cue/load#Config
|
||||
type TagMap map[string]*string
|
||||
|
||||
func (t TagMap) Tags() []string {
|
||||
parts := make([]string, 0, len(t))
|
||||
for k, v := range t {
|
||||
parts = append(parts, fmt.Sprintf("%s=%s", k, v))
|
||||
for tag, val := range t {
|
||||
if val == nil {
|
||||
parts = append(parts, tag)
|
||||
} else {
|
||||
parts = append(parts, fmt.Sprintf("%s=%s", tag, *val))
|
||||
}
|
||||
}
|
||||
return parts
|
||||
}
|
||||
@@ -60,10 +69,14 @@ func (t TagMap) String() string {
|
||||
// is not supported.
|
||||
func (t TagMap) Set(value string) error {
|
||||
parts := strings.SplitN(value, "=", 2)
|
||||
if len(parts) != 2 {
|
||||
switch len(parts) {
|
||||
case 1:
|
||||
t[parts[0]] = nil
|
||||
case 2:
|
||||
t[parts[0]] = &parts[1]
|
||||
default:
|
||||
return errors.Format("invalid format, must be tag=value")
|
||||
}
|
||||
t[parts[0]] = parts[1]
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -307,6 +320,7 @@ type BuildOpts struct {
|
||||
Stderr io.Writer
|
||||
WriteTo string
|
||||
Path string
|
||||
Tags []string
|
||||
}
|
||||
|
||||
func NewBuildOpts(path string) BuildOpts {
|
||||
@@ -316,5 +330,6 @@ func NewBuildOpts(path string) BuildOpts {
|
||||
Stderr: os.Stderr,
|
||||
WriteTo: "deploy",
|
||||
Path: path,
|
||||
Tags: make([]string, 0, 10),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,10 +43,11 @@ func RunCmd(ctx context.Context, name string, args ...string) (result RunResult,
|
||||
cmd.Stdout = result.Stdout
|
||||
cmd.Stderr = result.Stderr
|
||||
log := logger.FromContext(ctx)
|
||||
log.DebugContext(ctx, "running: "+name, "name", name, "args", args)
|
||||
command := fmt.Sprintf("%s '%s'", name, strings.Join(args, "' '"))
|
||||
log.DebugContext(ctx, "running command: "+command, "name", name, "args", args)
|
||||
err = cmd.Run()
|
||||
if err != nil {
|
||||
err = fmt.Errorf("could not run command: %s %s: %w", name, strings.Join(args, " "), err)
|
||||
err = fmt.Errorf("could not run command:\n\t%s\n\t%w", command, err)
|
||||
}
|
||||
return result, err
|
||||
}
|
||||
|
||||
@@ -1 +1 @@
|
||||
0
|
||||
3
|
||||
|
||||
Reference in New Issue
Block a user