Previously the Helm generator had no support for the --kube-version
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Previously the Helm generator had no support for the --api-versions
flag. This is a problem for helm charts that conditionally render
resources based on this capability.
This patch plumbs support through the author and core schemas with a new
field similar to how the enable hooks field is handled.
Previously the holos command line expected a Platform and BuildPlan
resource at the top level of the exported data from CUE. This forced us
to use hidden fields for everything else.
This patch modifies the BuildData struct to first look for a holos top
level field and use it if present. This opens up other top level fields
for use by end users.
Our intent is to reserve any top level field prefixed with holos.
Note this follows how Timoni works as well.
This patch strips down the v1alpha4 core and author schemas to only with
is absolutely necessary for all holos users. Aspects of platform
configuration applicable to some, even most, but not all users will be
moved into documentation topics organized as a recipe book.
The functionality removed from the v1alpha4 author schemas in v1alpha5
will move into self contained examples documented as topics on the docs
site.
The overall purpose is to have a focused, composeable, maintainable
author schema to help people get started and ideally we can support for
years with making breaking changes.
With this patch the v1alpha5 helm guide test passes. We're not going to
have this guide anymore but it demonstrates we're back to where we were
with v1alpha4.
Cue uses --inject, -t as the flags to set variables for fields tagged
using @tag(var,type=string).
We used --tag, which is different and requires a mental mapping. Let's
use the same flag and also pass it multiple times like they require so
we can copy and paste the command line output from the debug logs into a
cue export command to see what's going on.
This patch deprecates the --cluster-name flag, use --inject
holos_cluster=mycluster instead.
This patch also removes the environment field from the Component core
API, leaving this to the user namespace to define via tags. We don't
want to be too opinionated on how users manage their platform, baking
environment into the schema is a slippery slope toward those kinds of
opinions.
Closes: #276
Without this patch holos render platform may hang until the overall
context timeout is reached. This is a problem because the user has no
idea why it's hung.
This patch adds a warning at the 5 second and another at the 10 second
mark indicating the lock may be deadlocked. The user can then remove
the directory.
Without this patch kustomize errors aren't surfaced when executing holos
render platform.
This patch gives a fighting chance to the user to figure out what's
going on. The stderr is copied, logged, and surfaced up to the parent
holos render platform command.
Previously the #Helm and #Kustomize build plan helpers were not defined
in the v1alpha4 Author API. We need this definition to update the
Quickstart guide for v1alpha4 from v1alpha3.
This patch defines the #Helm and #Kustomize helpers in the Author API
similar to how #Kubernetes is defined.
Without this patch the --write-to flag can't be controlled from the
PlatformSpec in the CoreAPI. We need to surface this for the ArgoConfig
struct in the AuthorAPI.
That is to say, in v1alpha3 the --write-to flag was previously assumed
to be deploy/ in ArgoConfig using the DeployFiles functionality. We no
longer have DeployFiles in Core API v1alpha4, all artifacts are instead
written relative to the --write-to flag. Still, we need to expose this
flag in the PlatformSpec so users can use something other than the
deploy directory.
Previously the file generator was unimplemented. This patch implements
it as a simple file read into the ArtifactMap for use by the Kustomize
or Join transformers.
With this patch all v1alpha4 Core API features are implemented.
Resources, Helm, and File generators. Kustomize and Join transformers.
Blank lines show up in the output which is confusing. This patch fixes
the only source location identified with the following command.
export HOLOS_LOG_LEVEL=debug
export HOLOS_LOG_FORMAT=json
holos render platform ./platform 2>&1 | jq -r 'select (.msg == "")'
Previously helm charts were cached only by name, which is a problem
because the wrong version would be used when previously cached.
This patch caches charts by name and version to ensure changes in the
version results in pulling the new cached version. It is the user's
responsibility to remove old versions.
This patch also ensures only a single go routine can run cacheChart() at
a time across processes. This is necessary when rendering a platform
because multiple processes will run the Helm generator concurrently, for
example when the same chart is used for multiple environments or
customers.
The mkdir system call serves as the locking mechanism, which is robust
and atomic on all commonly used file systems.
Previously the helm generator was not implemented and returned an error.
This patch is a first pass copying the helm method from
internal/render/helm.go
Basic testing performed with a podinfo chart. It works as the previous
versions in v1alpha3 and before works. This patch does not address the
cached version issue in #273
holos.FilePath is intended for paths relative to the platform root
directory. We use the Artifact to store lots of stuff not related to
the platform root directory, for example kustomization.yaml in a temp
dir. Most entries are not relative to the platform root directory given
the implicit cfg.WriteTo prefix.
Previously:
could not run: could not build dev-join: could not get foo.yaml: not set at builder/v1alpha4/builder.go:180
This is confusing because set has nothing to do with the missing input
from the cue code the user writes.
Result:
could not run: could not build test-join: missing foo.yaml at builder/v1alpha4/builder.go:180
This is better because it at doesn't distract the user from the fact
they're missing a foo.yaml generator output to align with the
transformer input.
The code was inlined in a number of places, it makes sense to move it to
the interface. It'll also make it easier to test, we can provide a null
writer concrete value.
Previously the Artifact collection was processed sequentially. This
patch provides a modest performance improvement, about 16% faster for
our simple 2 artifact use case, by processing each artifact
concurrently.
Platform rendering provides poor user feedback:
```
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
```
We want to see the metadata.name field of each BuildPlan. This patch
injects the build plan name from the platform spec to make the name
available through the end to end platform rendering process.
Result:
```
❯ holos render platform ./platforms/minimal
rendered stage-namespaces for cluster local in 146.078375ms
rendered prod-namespaces for cluster local in 146.544583ms
rendered test-namespaces for cluster local in 147.0535ms
rendered dev-namespaces for cluster local in 147.499166ms
rendered platform in 147.553875ms
```
With this patch the first use case of CUE Resources + Kustomize is fully
working, artifacts are written into the deploy directory.
❯ holos render platform ./platforms/minimal
rendered namespaces for cluster local in 143.068583ms
rendered namespaces for cluster local in 143.861834ms
rendered namespaces for cluster local in 144.072666ms
rendered namespaces for cluster local in 144.219417ms
rendered platform in 144.326625ms
The output indicates we need to plumb the BuildPlan metadata.name from
the PlatfromSpec through to the render component command. This is
necessary so we can report the correct name instead of just the base
path.
Without this patch holos writes a single yaml document that is a list.
It needs to write a file that contains multiple documents, each document
a map[string]any representing the kubernetes resource.
This patch fixes the problem. With this patch kustomize fully executes.
This patch implements the v1alpha4 component rendering builder for a
component BuildPlan. We don't yet have the CUE definitions, so this
hasn't been end to end tested yet, the next step is defining the
generators and transforms in the core API BuildPlan.
This patch plumbs the switch statement to branch on a v1alpha4
BuildPlan. Tags need to be passed from the render platform subcommand
to the render component subcommand via the --tags argument.
This patch implements minimal rendering of a v1alpha4 platform using the
new render.Builder interface.
Tags aren't wired up yet, but this patch does cleanly separate Builder
interface from the Artifacts. Platform rendering doesn't have an
artifact itself, all artifacts are produced by rendering each component,
so we'll see how that works when we make the same changes to component
rendering, breaking it down to a render.Builder interface that sets
values in an Artifact.
The holos cli does not use an interface to handle different Platform api
versions. This makes it difficult to evolve the API in a backwards
compatible way.
This patch adds a top level switch statement to the `holos render
platform` command. The switch discriminates on the Platform API
version. v1alpha3 and earlier are classified as legacy versions and
will use the existing strict types. v1alpha4 and later versions will
use an interface to render the platform, allowing for multiple types to
implement the platform rendering interface.
Without this patch deployments to the dev environment are failing with
the following error when commits are pushed to the main branch.
GIT_DETAIL=v0.93.0-3-g4db3fb4 GIT_SUFFIX= bash ./hack/deploy-dev
Cloning into 'holos-infra'...
could not validate
could not run: could not validate invalid BuildPlan: apiVersion invalid: want: v1alpha3 have: v1alpha2 at internal/builder/builder.go:308
could not run: could not render component: exit status 1 at internal/render/platform.go:48
make: *** [Makefile:147: dev-deploy] Error 1
This patch removes the api version check in the build plan validation
function. In the future, we should pass an interface internally in the
holos executable.
The result is holos render platform ./platform succeeds with this patch
applied.
Previously CUE paniced when holos tried to unify values originating from
two different cue runtimes. This patch fixes the problem by
initializaing cue.Value structs from the same cue context.
Log messages are also improved after making one complete pass through
the Try Holos Locally guide.
Now that we have multi-platform images, we need a way to easily deploy
them. This involves changing the image tag. kustomize edit is often
used to bump image tags, but we can do better providing it directly in
the unified CUE configuration.
This patch modifies the builder to unify user data *.json files
recursively under userdata/ into the #UserData definition of the holos
entrypoint.
This is to support automation that writes simple json files to version
control, executes holos render platform, then commits and pushes the
results for git ops to take over deployment.
The make deploy target is the reason this change exists, to demonstrate
how to automatically deploy a new container image.
This patch addresses Nate's feedback that it's difficult to know what
platform is being operated on.
Previously it wasn't clear where the platform id used for push and pull
comes from. The source of truth is the platform.metadata.json file
created when the platform is first generated using `holos generate
platform k3d`.
This patch removes the platformId field from the platform.config.json
file, renames the platform.config.json file to platform.model.json and
renames the internal symbols to match the domain language of "Platform
Model" instead of the less clear "config"
This patch also changes the API between holos and CUE to use the proto
json imported from the proto file instead of generated from the go code
generated from the proto file. The purpose is to ensure protojson
encoding is used end to end.
Default log handler:
The patch also changes the default log output to print only the message
to stderr. This addresses similar feedback from both Gary and Nate that
the output is skipped over because it feels like internal debug logs.
We still want 100% of output to go through the logger so we can ensure
each line can be made into valid json. Info messages however are meant
for the user and all other attributes can be stripped off by default.
If additional source location is necessary, enable the text or json
output format.
Protobuf JSON:
This patch modifies the API contract between holos and CUE to ensure
data is exchanged exclusively using protojson. This is necessary
because protobuf has a canonical json format which is not compatible
with the go json package struct tags. When Holos handles a protobuf
message, it must marshal and unmarshal it using the protojson package.
Similarly, when importing protobuf messages into CUE, we must use `cue
import` instead of `cue go get` so that the canonical format is used
instead of the invalid go json struct tags.
Finally, when a Go struct like v1alpha1.Form is used to represent data
defined in cue which contains a nested protobuf message, Holos should
use a cue.Value to lookup the nested path, marshal it into json bytes,
then unmarshal it again using protojson.
Previously a couple of methods were defined on the Result struct.
This patch moves the methods to an internal wrapper struct to remove
them from the API documentation.
With this patch the API between holos and CUE is entirely a data API.
With this patch the eso-creds-manager component renders correctly. This
is a `#Kubernetes` type build plan which uses the
spec.components.resources map to manage resources.
The only issue was needing to provide the namespace to the nested holos
component inside the BuildPlan.
The ArgoCD Application resource moves to the DeployFiles field of a
separate holos component in the same build plan at
spec.components.resources.argocd. For this reason a separate Result
object is no longer necessary inside of the Holos cli for the purpose of
managing Flux or ArgoCD gitops. The CUE code can simply inline whatever
gitops resources it wants and the holos cli will write the files
relative to the cluster specific deploy directory.
Result:
```
❯ holos render component --cluster-name management components/eso-creds-manager
2:55PM INF result.go:195 wrote deploy file version=0.84.1 path=deploy/clusters/management/gitops/eso-creds-manager.application.gen.yaml bytes=350
2:55PM INF render.go:92 rendered eso-creds-manager version=0.84.1 cluster=management name=eso-creds-manager status=ok action=rendered
```
Previously methods were defined on the API objects in the v1alpha1 API.
The API should be data structures only. This patch refactors the
methods responsible for orchestrating the build plan to pull them into
the internal render package.
The result is the API is cleaner and has no methods. The render package
has corresponding data structures which simply wrap around the API
structure and implement the methods to render and return the result to
the CLI.
This commit compiles, but it has not been tested at all. It's almost
surely broken completely.
Previously in v1alpha1, all Holos structs are located in the same
package. This makes it difficult to focus on only the structs necessary
to transfer configuration data from CUE to the `holos` cli.
This patch splits the structs into `meta` and `core` where the core
package holds the structs end users should refer to and focus on. Only
the Platform resource is in core now, but other BuildPlan types will be
added shortly.
Previously, each BuildPlan has no clear way to produce an ArgoCD
Application resource. This patch provides a general solution where each
BuildPlan can provide arbitrary files as a map[string]string where the
key is the file path relative to the gitops repository `deploy/` folder.
The first thing most platforms need to do is come up with a strategy for
managing namespaces across multiple clusters.
This patch defines #Namespaces in the holos platform and adds a
namespaces component which loops over all values in the #Namespaces
struct and manages a kubernetes Namespace object.
The platform resource itself loops over all clusters in the platform to
manage all namespaces across all clusters.
From a blank slate:
```
❯ holos generate platform holos
4:26PM INF platform.go:79 wrote platform.metadata.json version=0.82.0 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=/home/jeff/workspace/holos-run/holos-infra/saas/platform.metadata.json
4:26PM INF platform.go:91 generated platform holos version=0.82.0 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=/home/jeff/workspace/holos-run/holos-infra/saas
❯ holos pull platform config .
4:26PM INF pull.go:64 pulled platform model version=0.82.0 server=https://jeff.app.dev.k2.holos.run:443 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc
4:26PM INF pull.go:75 saved platform config version=0.82.0 server=https://jeff.app.dev.k2.holos.run:443 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=platform.config.json
❯ (cd components && holos generate component cue namespaces)
4:26PM INF component.go:147 generated component version=0.82.0 name=namespaces path=/home/jeff/workspace/holos-run/holos-infra/saas/components/namespaces
❯ holos render platform ./platform/
4:26PM INF platform.go:29 ok render component version=0.82.0 path=components/namespaces cluster=management num=1 total=2 duration=464.055541ms
4:26PM INF platform.go:29 ok render component version=0.82.0 path=components/namespaces cluster=aws1 num=2 total=2 duration=467.978499ms
```
The result:
```sh
cat deploy/clusters/management/components/namespaces/namespaces.gen.yaml
```
```yaml
---
metadata:
name: holos
labels:
kubernetes.io/metadata.name: holos
kind: Namespace
apiVersion: v1
```
Split holos render into component and platform.
This patch splits the previous `holos render` command into subcommands.
`holos render component ./path/to/component/` behaves as the previous
`holos render` command and renders an individual component.
The new `holos render platform ./path/to/platform/` subcommand makes
space to render the entire platform using the platform model pulled from
the PlatformService.
Starting with an empty directory:
```sh
holos register user
holos generate platform bare
holos pull platform config .
holos render platform ./platform/
```
```txt
10:01AM INF platform.go:29 ok render component version=0.80.2 path=components/configmap cluster=k1 num=1 total=1 duration=448.133038ms
```
The bare platform has a single component which refers to the platform
model pulled from the PlatformService:
```sh
cat deploy/clusters/mycluster/components/platform-configmap/platform-configmap.gen.yaml
```
```yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
name: platform
namespace: default
data:
platform: |
spec:
model:
cloud:
providers:
- cloudflare
cloudflare:
email: platform@openinfrastructure.co
org:
displayName: Open Infrastructure Services
name: ois
```
This patch adds a subcommand to pull the data necessary to construct a
PlatformConfig DTO. The PlatformConfig message contains all of the
fields and values necessary to build a platform and the platform
components. This is an alternative to holos passing multiple tags to
CUE. The PlatformConfig is marshalled and passed once.
The platform config is also stored in the local filesystem in the root
directory of the platform. This enables repeated local building and
rendering without making an rpc call.
The build / render pipeline is expected to cache the PlatformConfig once
at the start of the pipeline using the pull subcommand.
