Compare commits

...

3 Commits

Author SHA1 Message Date
Jeff McCune
77836be250 docs: update readme 2024-11-25 08:10:11 -08:00
Jeff McCune
4db670b854 docs: add validators tutorial (#357)
Add a tutorial page on validators.
2024-11-24 21:34:09 -08:00
Jeff McCune
d87c919519 docs: redirect /docs/topics to structures
/docs/v1alpha5/api/author/ links to it in the opening paragraphs.
2024-11-24 19:38:30 -08:00
9 changed files with 554 additions and 39 deletions

129
README.md
View File

@@ -1,35 +1,130 @@
## Holos - A Holistic Development Platform
# Holos
<img width="50%"
align="right"
style="display: block; margin: 40px auto;"
src="https://openinfrastructure.co/blog/2016/02/27/logo/logorectangle.png">
Building and maintaining a software development platform is a complex and time
consuming endeavour. Organizations often dedicate a team of 3-4 who need 6-12
months to build the platform.
[Holos] is a configuration management tool for Kubernetes implementing the
[rendered manifests pattern]. It handles configurations ranging from single
resources to multi-cluster platforms across regions.
Holos is a tool and a reference platform to reduce the complexity and speed up
the process of building a modern, cloud native software development platform.
Key components:
- Platform schemas defining component integration
- Building blocks unifying Helm, Kustomize and Kubernetes configs with CUE
- BuildPlan pipeline for generating, transforming and validating manifests
- **Accelerate new projects** - Reduce time to market and operational complexity by starting your new project on top of the Holos reference platform.
- **Modernize existing projects** - Incrementally incorporate your existing platform services into Holos for simpler integration.
- **Unified configuration model** - Increase safety and reduce the risk of config changes with CUE.
- **First class Helm and Kustomize support** - Leverage and reuse your existing investment in existing configuration tools such as Helm and Kustomize.
- **Modern Authentication and Authorization** - Holos seamlessly integrates platform identity and access management with zero-trust beyond corp style authorization policy.
```mermaid
---
title: Rendering Overview
---
graph LR
Platform[<a href="https://holos.run/docs/v1alpha5/api/author/#Platform">Platform</a>]
Component[<a href="https://holos.run/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
## Quick Installation
Helm[<a href="https://holos.run/docs/v1alpha5/api/author/#Helm">Helm</a>]
Kustomize[<a href="https://holos.run/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
Kubernetes[<a href="https://holos.run/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
```console
go install github.com/holos-run/holos/cmd/holos@latest
BuildPlan[<a href="https://holos.run/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
ResourcesArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
GitOpsArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
Generators[<a href="https://holos.run/docs/v1alpha5/api/core/#Generator">Generators</a>]
Transformers[<a href="https://holos.run/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
Validators[<a href="https://holos.run/docs/v1alpha5/api/core/#Validator">Validators</a>]
Files[Manifest<br/>Files]
Platform --> Component
Component --> Helm --> BuildPlan
Component --> Kubernetes --> BuildPlan
Component --> Kustomize --> BuildPlan
BuildPlan --> ResourcesArtifact --> Generators
BuildPlan --> GitOpsArtifact --> Generators
Generators --> Transformers --> Validators --> Files
```
## Docs and Support
## Setup
The documentation for developing and using Holos is available at: https://holos.run
```shell
brew install holos-run/tap/holos
```
For discussion and support, [open a discussion](https://github.com/orgs/holos-run/discussions/new/choose).
Refer to [setup] for other installation methods and dependencies.
## Example
See our [tutorial] for a complete hello world example.
```cue showLineNumbers
package holos
holos: Component.BuildPlan
Component: #Helm & {
Name: "podinfo"
Chart: {
version: "6.6.2"
repository: {
name: "podinfo"
url: "https://stefanprodan.github.io/podinfo"
}
}
Values: ui: {
message: string | *"Hello World" @tag(message, type=string)
}
}
```
## Organizational Role
Platform engineers use Holos to generate Kubernetes manifests, both locally and
in CI pipelines. The manifests are committed to version control and deployed via
GitOps tools like ArgoCD or Flux.
Holos integrates seamlessly with existing Helm charts, Kustomize bases, and
other version-controlled configurations.
## Advantages of Holos
### Safe
Holos leverages [CUE] for strong typing and validation of configuration data,
ensuring consistent output from Helm and other tools.
### Consistent
A unified pipeline processes all configurations - whether from CUE, Helm, or
Kustomize - through the same well-defined stages.
### Flexible
Composable building blocks for generation, transformation, validation and
integration let teams assemble workflows that match their needs.
The core is intentionally unopinionated about platform configuration patterns.
Common needs like environments and clusters are provided as customizable
[topics] recipes rather than enforced structures.
## Getting Help
Get support through our [Discord] channel or [GitHub discussions]. Configuration
challenges arise at all experience levels - we welcome your questions and are
here to help.
## License
Holos is licensed under Apache 2.0 as found in the [LICENSE file](LICENSE).
[Holos]: https://holos.run
[rendered manifests pattern]: https://akuity.io/blog/the-rendered-manifests-pattern
[CUE]: https://cuelang.org/
[Discord]: https://discord.gg/JgDVbNpye7
[GitHub discussions]: https://github.com/holos-run/holos/discussions
[Why CUE for Configuration]: https://holos.run/blog/why-cue-for-configuration/
[topics]: https://holos.run/docs/topics/
[setup]: https://holos.run/docs/setup/
[tutorial]: https://holos.run/docs/tutorial/

View File

@@ -209,7 +209,10 @@ type FilePath string
type FileContent string
// Validator validates files. Useful to validate an [Artifact] prior to writing
// it out to the final destination. Validators may be executed concurrently.
// it out to the final destination. Holos may execute validators concurrently.
// See the [validators] tutorial for an end to end example.
//
// [validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
type Validator struct {
// Kind represents the kind of transformer. Must be Kustomize, or Join.
Kind string `json:"kind" yaml:"kind" cue:"\"Command\""`

View File

@@ -418,7 +418,7 @@ type Transformer struct {
<a name="Validator"></a>
## type Validator {#Validator}
Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Validators may be executed concurrently.
Validator validates files. Useful to validate an [Artifact](<#Artifact>) prior to writing it out to the final destination. Holos may execute validators concurrently. See the [validators](<https://holos.run/docs/v1alpha5/tutorial/validators/>) tutorial for an end to end example.
```go
type Validator struct {

View File

@@ -265,9 +265,6 @@ This tutorial uses the `#Resources` structure to map resource kinds to their
schema definitions in CUE. This structure is defined in `resources.cue` at the
root of the tree. Take a look at [resources.cue] to see this mapping structure.
Continue to the next tutorial to learn how to define your own data structures
similar to the `#Resources` structure.
[Local Cluster]: ../topics/local-cluster.mdx
[ExternalSecret]: https://external-secrets.io/latest/api/externalsecret/
[Artifact]: ../api/core.md#Artifact

View File

@@ -11,7 +11,7 @@ import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
## Overview
Holos is a configuration management tool for Kubernetes that implements the
Holos is a configuration management tool for Kubernetes implementing the
[rendered manifests pattern]. It handles configurations ranging from single
resources to multi-cluster platforms across regions.

View File

@@ -0,0 +1,411 @@
---
slug: validators
title: Validators
description: Validate rendered manifests against policy definitions.
sidebar_position: 60
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RenderingOverview from '@site/src/diagrams/rendering-overview.mdx';
# Validators
## Overview
Sometimes Helm charts render Secrets we do not wanted committed to version
control for security. Helm charts often render incorrect manifests, even if
they're accepted by the api server. For example, passing `null` to collection
fields. We'll solve both of these issues using a [Validator] to block artifacts
with a Secret resource, and verifying the artifact against Kubernetes type
definitions.
1. If a Helm chart renders a Secret, Holos errors before writing the artifact
and suggests an ExternalSecret instead.
2. Each resource is validated against a field named by the value of the kind
field. For example, a `kind: Secret` resource validates against `secret: {}` in
CUE. `kind: Deployment` validates against `deployment: {}` in CUE.
3. The final artifact is validated, covering the output of all generators and
transformers.
<RenderingOverview />
## The Code
### Generating the Structure
Use `holos` to generate a minimal platform directory structure. First, create
and navigate into a blank directory. Then, use the `holos generate platform`
command to generate a minimal platform.
```shell
mkdir holos-validators-tutorial && cd holos-validators-tutorial
holos init platform v1alpha5
```
### Creating the Component
Create the directory for the `podinfo` component. Create an empty file, then add
the following CUE configuration to it.
```bash
mkdir -p components/podinfo
```
```bash
cat <<EOF > components/podinfo/podinfo.cue
```
```cue showLineNumbers
package holos
// export the component build plan to holos
holos: Component.BuildPlan
// Component is a Helm chart
Component: #Helm & {
Name: "podinfo"
Namespace: "default"
// Add metadata.namespace to all resources with kustomize.
KustomizeConfig: Kustomization: namespace: Namespace
Chart: {
version: "6.6.2"
repository: {
name: "podinfo"
url: "https://stefanprodan.github.io/podinfo"
}
}
}
```
```bash
EOF
```
Register the component with the platform.
```bash
cat <<EOF > platform/podinfo.cue
```
```cue showLineNumbers
package holos
Platform: Components: podinfo: {
name: "podinfo"
path: "components/podinfo"
}
```
```bash
EOF
```
Render the platform.
<Tabs groupId="tutorial-hello-render-manifests">
<TabItem value="command" label="Command">
```bash
holos render platform
```
</TabItem>
<TabItem value="output" label="Output">
```
cached podinfo 6.6.2
rendered podinfo in 1.938665041s
rendered platform in 1.938759417s
```
</TabItem>
</Tabs>
Add and commit the initial configuration.
```bash
git init . && git add . && git commit -m initial
```
### Define the Valid Schema
We'll use a CUE package named `policy` so the entire platform configuration in
package `holos` isn't loaded every time we validate an artifact.
Create `policy/validation-schema.cue` with the following content.
```shell
mkdir -p policy
cat <<EOF > policy/validation-schema.cue
```
```cue showLineNumbers
package policy
import apps "k8s.io/api/apps/v1"
// Block Secret resources. kind will not unify with "Secret"
secret: kind: "Use an ExternalSecret instead. Forbidden by security policy."
// Validate Deployment against Kubernetes type definitions.
deployment: apps.#Deployment
```
```shell
EOF
```
### Configuring Validators
Configure the Validators [ComponentConfig] field to configure each [BuildPlan]
to validate the rendered [Artifact] files.
```shell
cat <<EOF > validators.cue
```
```cue showLineNumbers
package holos
// Configure all component kinds to validate against the policy directory.
#ComponentConfig: Validators: cue: {
kind: "Command"
// Note --path maps each resource to a top level field named by the kind.
command: args: ["holos", "cue", "vet", "./policy", "--path", "strings.ToLower(kind)"]
}
```
```shell
EOF
```
### Patching Errors
Render the platform to see validation fail. The podinfo chart has no Secret,
but it produces an invalid Deployment because it sets the container resource
limits field to `null`.
```shell
holos render platform
```
```txt
deployment.spec.template.spec.containers.0.resources.limits: conflicting values null and {[string]:"k8s.io/apimachinery/pkg/api/resource".#Quantity} (mismatched types null and struct):
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:355:9
./cue.mod/gen/k8s.io/api/apps/v1/types_go_gen.cue:376:12
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2840:11
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:2968:14
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:15
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:3882:18
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:5027:9
./cue.mod/gen/k8s.io/api/core/v1/types_go_gen.cue:6407:16
./policy/validation-schema.cue:9:13
../../../../../var/folders/22/T/holos.validate1636392304/components/podinfo/podinfo.gen.yaml:104:19
could not run: terminating because of errors
could not run: could not validate podinfo path ./components/podinfo: could not run command: holos cue vet ./policy --path strings.ToLower(kind) /var/folders/22/T/holos.validate1636392304/components/podinfo/podinfo.gen.yaml: exit status 1 at builder/v1alpha5/builder.go:411
could not run: could not render component: could not run command: holos --log-level info --log-format console render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo: exit status 1 at cli/render/render.go:155
```
We'll use a [Kustomize] patch [Transformer] to replace the `null` limits field
with a valid equivalent value.
:::important
This configuration is defined in CUE, not YAML, even though we're configuring a
Kustomize patch transformer. CUE gives us access to the unified platform
configuration.
:::
```shell
cat <<EOF > components/podinfo/patch.cue
```
```cue showLineNumbers
package holos
import "encoding/yaml"
Component: KustomizeConfig: Kustomization: {
_patches: limits: {
target: kind: "Deployment"
patch: yaml.Marshal([{
op: "test"
path: "/spec/template/spec/containers/0/resources/limits"
value: null
}, {
op: "replace"
path: "/spec/template/spec/containers/0/resources/limits"
value: {}
}])
}
patches: [for x in _patches {x}]
}
```
```shell
EOF
```
Now the platform renders.
<Tabs groupId="3A050092-8E56-49D4-84A9-71E544A21276">
<TabItem value="command" label="Command">
```bash
holos render platform
```
</TabItem>
<TabItem value="output" label="Output">
```txt
rendered podinfo in 181.875083ms
rendered platform in 181.975833ms
```
</TabItem>
</Tabs>
## Inspecting the BuildPlan
The BuildPlan patches the output of the upstream helm chart without modifying
it, then validates the artifact against the Kubernetes type definitions.
<Tabs groupId="1DAB4C46-0793-4CCA-8930-7B2E60BDA1BE">
<TabItem value="command" label="Command">
```bash
holos show buildplans
```
</TabItem>
<TabItem value="output" label="Output">
```yaml showLineNumbers
kind: BuildPlan
apiVersion: v1alpha5
metadata:
name: podinfo
spec:
artifacts:
- artifact: components/podinfo/podinfo.gen.yaml
generators:
- kind: Helm
output: helm.gen.yaml
helm:
chart:
name: podinfo
version: 6.6.2
release: podinfo
repository:
name: podinfo
url: https://stefanprodan.github.io/podinfo
values: {}
namespace: default
- kind: Resources
output: resources.gen.yaml
transformers:
- kind: Kustomize
inputs:
- helm.gen.yaml
- resources.gen.yaml
output: components/podinfo/podinfo.gen.yaml
kustomize:
kustomization:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
patches:
- patch: |
- op: test
path: /spec/template/spec/containers/0/resources/limits
value: null
- op: replace
path: /spec/template/spec/containers/0/resources/limits
value: {}
target:
kind: Deployment
name: ""
resources:
- helm.gen.yaml
- resources.gen.yaml
validators:
- kind: Command
inputs:
- components/podinfo/podinfo.gen.yaml
command:
args:
- holos
- cue
- vet
- ./policy
- --path
- strings.ToLower(kind)
```
</TabItem>
</Tabs>
## Catching Mistakes
Suppose a teammate downloads a helm chart that includes a Secret unbeknown to
them. Holos catches the problem and suggests an ExternalSecret instead.
Mix in a Secret to see what happens
```shell
cat <<EOF > components/podinfo/secret.cue
```
```cue showLineNumbers
package holos
Component: Resources: Secret: example: metadata: name: "example"
```
```shell
EOF
```
Render the platform to see the error.
```shell
holos render platform
```
```txt
secret.kind: conflicting values "Use an ExternalSecret instead. Forbidden by security policy." and "Secret":
./policy/validation-schema.cue:6:15
../../../../../var/folders/22/T/holos.validate2549739170/components/podinfo/podinfo.gen.yaml:1:7
could not run: terminating because of errors
could not run: could not validate podinfo path ./components/podinfo: could not run command: holos cue vet ./policy --path strings.ToLower(kind) /var/folders/22/T/holos.validate2549739170/components/podinfo/podinfo.gen.yaml: exit status 1 at builder/v1alpha5/builder.go:411
could not run: could not render component: could not run command: holos --log-level info --log-format console render component --inject holos_component_name=podinfo --inject holos_component_path=components/podinfo ./components/podinfo: exit status 1 at cli/render/render.go:155
```
:::important
Holos quickly returns an error if validated artifacts have a Secret.
:::
Remove the secret to resolve the issue.
```shell
rm components/podinfo/secret.cue
```
## Inspecting the diff
The validation and patch results in a correct Deployment, verified against the
Kubernetes type definitions.
```shell
git diff
```
```diff
diff --git a/deploy/components/podinfo/podinfo.gen.yaml b/deploy/components/podinfo/podinfo.gen.yaml
index 6e4aec0..a145e3f 100644
--- a/deploy/components/podinfo/podinfo.gen.yaml
+++ b/deploy/components/podinfo/podinfo.gen.yaml
@@ -101,7 +101,7 @@ spec:
successThreshold: 1
timeoutSeconds: 5
resources:
- limits: null
+ limits: {}
requests:
cpu: 1m
memory: 16Mi
```
## Trying Locally
Optionally, apply the manifests rendered by Holos to a [Local Cluster] for
testing.
[Local Cluster]: ../topics/local-cluster.mdx
[ExternalSecret]: https://external-secrets.io/latest/api/externalsecret/
[Artifact]: ../api/core.md#Artifact
[BuildPlan]: ../api/core.md#BuildPlan
[Resources]: ../api/core.md#Resources
[Validator]: ../api/core.md#Validator
[Transformer]: ../api/core.md#Transformer
[Kustomize]: ../api/core.md#Kustomize
[Generator]: ../api/core.md#Generator
[Hello Holos]: ./hello-holos.mdx
[cue.mod/gen/external-secrets.io/externalsecret/v1beta1/types_gen.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/cue.mod/gen/external-secrets.io/externalsecret/v1beta1/types_gen.cue#L13
[ComponentConfig]: ../api/author.md#ComponentConfig
[timoni]: https://timoni.sh/install/
[resources.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/v1alpha5/resources.cue#L33

View File

@@ -3,21 +3,21 @@
title: Rendering Overview
---
graph LR
Platform[<a href="/docs/v1alpha5/api/author/#Platform">Platform</a>]
Component[<a href="/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
Platform[<a href="https://holos.run/docs/v1alpha5/api/author/#Platform">Platform</a>]
Component[<a href="https://holos.run/docs/v1alpha5/api/author/#ComponentConfig">Components</a>]
Helm[<a href="/docs/v1alpha5/api/author/#Helm">Helm</a>]
Kustomize[<a href="/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
Kubernetes[<a href="/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
Helm[<a href="https://holos.run/docs/v1alpha5/api/author/#Helm">Helm</a>]
Kustomize[<a href="https://holos.run/docs/v1alpha5/api/author/#Kustomize">Kustomize</a>]
Kubernetes[<a href="https://holos.run/docs/v1alpha5/api/author/#Kubernetes">Kubernetes</a>]
BuildPlan[<a href="/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
BuildPlan[<a href="https://holos.run/docs/v1alpha5/api/core/#BuildPlan">BuildPlan</a>]
ResourcesArtifact[<a href="/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
GitOpsArtifact[<a href="/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
ResourcesArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">Resources<br/>Artifact</a>]
GitOpsArtifact[<a href="https://holos.run/docs/v1alpha5/api/core/#Artifact">GitOps<br/>Artifact</a>]
Generators[<a href="/docs/v1alpha5/api/core/#Generator">Generators</a>]
Transformers[<a href="/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
Validators[Validators]
Generators[<a href="https://holos.run/docs/v1alpha5/api/core/#Generator">Generators</a>]
Transformers[<a href="https://holos.run/docs/v1alpha5/api/core/#Transformer">Transformers</a>]
Validators[<a href="https://holos.run/docs/v1alpha5/api/core/#Validator">Validators</a>]
Files[Manifest<br/>Files]
Platform --> Component

View File

@@ -1,4 +1,10 @@
/docs /docs/v1alpha5/ 301
/docs/ /docs/v1alpha5/ 301
/docs/overview /docs/v1alpha5/tutorial/overview/ 301
/docs/overview/ /docs/v1alpha5/tutorial/overview/ 301
/docs /docs/v1alpha5/ 301
/docs/ /docs/v1alpha5/ 301
/docs/tutorial /docs/v1alpha5/tutorial/ 301
/docs/tutorial/ /docs/v1alpha5/tutorial/ 301
/docs/overview /docs/v1alpha5/tutorial/overview/ 301
/docs/overview/ /docs/v1alpha5/tutorial/overview/ 301
/docs/topics /docs/v1alpha5/topics/structures/ 301
/docs/topics/ /docs/v1alpha5/topics/structures/ 301
/docs/setup /docs/v1alpha5/tutorial/setup/ 301
/docs/setup/ /docs/v1alpha5/tutorial/setup/ 301

View File

@@ -232,7 +232,10 @@ package core
#FileContent: string
// Validator validates files. Useful to validate an [Artifact] prior to writing
// it out to the final destination. Validators may be executed concurrently.
// it out to the final destination. Holos may execute validators concurrently.
// See the [validators] tutorial for an end to end example.
//
// [validators]: https://holos.run/docs/v1alpha5/tutorial/validators/
#Validator: {
// Kind represents the kind of transformer. Must be Kustomize, or Join.
kind: string & "Command" @go(Kind)