This patch strips down the v1alpha4 core and author schemas to only with
is absolutely necessary for all holos users. Aspects of platform
configuration applicable to some, even most, but not all users will be
moved into documentation topics organized as a recipe book.
The functionality removed from the v1alpha4 author schemas in v1alpha5
will move into self contained examples documented as topics on the docs
site.
The overall purpose is to have a focused, composeable, maintainable
author schema to help people get started and ideally we can support for
years with making breaking changes.
With this patch the v1alpha5 helm guide test passes. We're not going to
have this guide anymore but it demonstrates we're back to where we were
with v1alpha4.
It's too long for documentation. Shorten it for clarity.
Result:
```
❯ holos render platform ./platform
rendered bank-accounts-db for cluster workload in 160.7245ms
rendered bank-ledger-db for cluster workload in 162.465625ms
rendered bank-userservice for cluster workload in 166.150417ms
rendered bank-ledger-writer for cluster workload in 168.075459ms
rendered bank-balance-reader for cluster workload in 172.492292ms
rendered bank-backend-config for cluster workload in 198.117916ms
rendered bank-secrets for cluster workload in 223.200042ms
rendered gateway for cluster workload in 124.841917ms
rendered httproutes for cluster workload in 131.86625ms
rendered bank-contacts for cluster workload in 154.463792ms
rendered bank-transaction-history for cluster workload in 159.968208ms
rendered bank-frontend for cluster workload in 325.24425ms
rendered app-projects for cluster workload in 110.577916ms
rendered ztunnel for cluster workload in 137.502792ms
rendered cni for cluster workload in 209.993375ms
rendered cert-manager for cluster workload in 172.933834ms
rendered external-secrets for cluster workload in 135.759792ms
rendered local-ca for cluster workload in 98.026708ms
rendered istiod for cluster workload in 403.050833ms
rendered argocd for cluster workload in 294.663167ms
rendered gateway-api for cluster workload in 228.47875ms
rendered namespaces for cluster workload in 113.586916ms
rendered base for cluster workload in 533.76675ms
rendered external-secrets-crds for cluster workload in 529.053375ms
rendered crds for cluster workload in 931.180458ms
rendered platform in 1.248310167s
```
Previously:
```
❯ holos render platform ./platform
rendered projects/bank-of-holos/backend/components/bank-ledger-db for cluster workload in 158.534875ms
rendered projects/bank-of-holos/backend/components/bank-accounts-db for cluster workload in 159.836166ms
rendered projects/bank-of-holos/backend/components/bank-userservice for cluster workload in 160.360667ms
rendered projects/bank-of-holos/backend/components/bank-balance-reader for cluster workload in 169.478584ms
rendered projects/bank-of-holos/backend/components/bank-ledger-writer for cluster workload in 169.437833ms
rendered projects/bank-of-holos/backend/components/bank-backend-config for cluster workload in 182.089333ms
rendered projects/bank-of-holos/security/components/bank-secrets for cluster workload in 196.502792ms
rendered projects/platform/components/istio/gateway for cluster workload in 122.273083ms
rendered projects/bank-of-holos/frontend/components/bank-frontend for cluster workload in 307.573584ms
rendered projects/platform/components/httproutes for cluster workload in 149.631583ms
rendered projects/bank-of-holos/backend/components/bank-contacts for cluster workload in 153.529708ms
rendered projects/bank-of-holos/backend/components/bank-transaction-history for cluster workload in 165.375667ms
rendered projects/platform/components/app-projects for cluster workload in 107.253958ms
rendered projects/platform/components/istio/ztunnel for cluster workload in 137.22275ms
rendered projects/platform/components/istio/cni for cluster workload in 233.980958ms
rendered projects/platform/components/cert-manager for cluster workload in 171.966958ms
rendered projects/platform/components/external-secrets for cluster workload in 134.207792ms
rendered projects/platform/components/istio/istiod for cluster workload in 403.19ms
rendered projects/platform/components/local-ca for cluster workload in 97.544708ms
rendered projects/platform/components/argocd/argocd for cluster workload in 289.577208ms
rendered projects/platform/components/gateway-api for cluster workload in 218.290458ms
rendered projects/platform/components/namespaces for cluster workload in 109.534125ms
rendered projects/platform/components/istio/base for cluster workload in 526.32525ms
rendered projects/platform/components/external-secrets-crds for cluster workload in 523.7495ms
rendered projects/platform/components/argocd/crds for cluster workload in 1.002546375s
rendered platform in 1.312824333s
```
This patch adds Istio to the Expose a Service documentation and
introduces new concepts. The Kubernetes build plan schema, the
namespaces component, and an example of how to safely re-use Helm values
from the root to multiple leaf components.
fix: istio cni not ready on k3d
---
The istio-k3d component embedded into holos fixes the cni pod not
becoming ready with our k3d local cluster guide. The pod log error this
fixes is:
configuration requires updates, (re)writing CNI config file at "": no networks found in /host/etc/cni/net.d
Istio CNI is configured as chained plugin, but cannot find existing CNI network config: no networks found in /host/etc/cni/net.d
Waiting for CNI network config file to be written in /host/etc/cni/net.d...
[Platform k3d]: https://istio.io/latest/docs/ambient/install/platform-prerequisites/#k3d
docs: clarify how to reset the local cluster
---
This is something we do all the time while developing and documenting,
so make it easy and fast to reset the cluster to a known good state.
Previously helm and cue components were split into two different
subcommands off the holos generate component command. This is
unnecessary, I'm not sure why it was there in the first place. The code
seemed perfectly duplicated.
This patch combines them to focus on the concept of a Component. It
doesn't matter what kind it is now that it's expected to be run from the
root of the platform repository and drop configuration at the root and
the leaf of the tree.
Make sure go install works from the quickstart documentation by doing a
release. Otherwise, v0.93.1 is installed which doesn't include the
platform schema.
Without this patch the `holos generate platform` command automatically
makes an rpc call to holos server. This creates friction for the
quickstart guide because we don't need to require users to register and
have an organization and platform already created in the server just to
generate a simple platform to exercise a simple helm chart component.
A future patch should implement the behavior of linking a server side
platform to a local git repository by making the API call to get the
platform ID then updating the platform.metadata.json file.
Previously CUE paniced when holos tried to unify values originating from
two different cue runtimes. This patch fixes the problem by
initializaing cue.Value structs from the same cue context.
Log messages are also improved after making one complete pass through
the Try Holos Locally guide.
This patch addresses Nate's feedback that it's difficult to know what
platform is being operated on.
Previously it wasn't clear where the platform id used for push and pull
comes from. The source of truth is the platform.metadata.json file
created when the platform is first generated using `holos generate
platform k3d`.
This patch removes the platformId field from the platform.config.json
file, renames the platform.config.json file to platform.model.json and
renames the internal symbols to match the domain language of "Platform
Model" instead of the less clear "config"
This patch also changes the API between holos and CUE to use the proto
json imported from the proto file instead of generated from the go code
generated from the proto file. The purpose is to ensure protojson
encoding is used end to end.
Default log handler:
The patch also changes the default log output to print only the message
to stderr. This addresses similar feedback from both Gary and Nate that
the output is skipped over because it feels like internal debug logs.
We still want 100% of output to go through the logger so we can ensure
each line can be made into valid json. Info messages however are meant
for the user and all other attributes can be stripped off by default.
If additional source location is necessary, enable the text or json
output format.
Protobuf JSON:
This patch modifies the API contract between holos and CUE to ensure
data is exchanged exclusively using protojson. This is necessary
because protobuf has a canonical json format which is not compatible
with the go json package struct tags. When Holos handles a protobuf
message, it must marshal and unmarshal it using the protojson package.
Similarly, when importing protobuf messages into CUE, we must use `cue
import` instead of `cue go get` so that the canonical format is used
instead of the invalid go json struct tags.
Finally, when a Go struct like v1alpha1.Form is used to represent data
defined in cue which contains a nested protobuf message, Holos should
use a cue.Value to lookup the nested path, marshal it into json bytes,
then unmarshal it again using protojson.
The github workflows fail because yarn is not available. The Angular
frontend app uses npm so we should also use npm for the website to
minimize dependencies.
Previously `go install` fails to install holos.
```
❯ go install github.com/holos-run/holos/cmd/holos@latest
../../go/pkg/mod/github.com/holos-run/holos@v0.86.0/internal/frontend/frontend.go:25:12: pattern holos/dist/holos/ui/index.html: no matching files found
../../go/pkg/mod/github.com/holos-run/holos@v0.86.0/doc/website/website.go:14:12: pattern all:build: no matching files found
```
This is because we do not commit required files. This patch fixes the
problem by following Rob Pike's guidance to commit generated files.
This patch also replaces the previous use of Makefile tasks to generate
code with //go:generate directives.
This means the process of keeping the source code clean is straight
forward:
```
git clone
make tools
make generate
make build
```
Refer to https://go.dev/blog/generate
> Also, if the containing package is intended for import by go get, once
> the file is generated (and tested!) it must be checked into the source
> code repository to be available to clients. - Rob Pike
Previously docs are not published. This patch adds Docusaurus into the
doc/website directory which is also a Go package to embed the static
site into the executable.
Serve the site using http.Server with a h2c handler with the command:
holos website --log-format=json --log-drop=source
The website subcommand is intended to be run from a container as a
Deployment. For expedience, the website subcommand doesn't use the
signals package like the server subcommand does. Consider using it for
graceful Deployment restarts.
Refer to https://github.com/ent/ent/tree/master/doc/website
Previously a couple of methods were defined on the Result struct.
This patch moves the methods to an internal wrapper struct to remove
them from the API documentation.
With this patch the API between holos and CUE is entirely a data API.
Previously, each BuildPlan has no clear way to produce an ArgoCD
Application resource. This patch provides a general solution where each
BuildPlan can provide arbitrary files as a map[string]string where the
key is the file path relative to the gitops repository `deploy/` folder.
Use v0.81.2 to build out the holos platform. Once we have the
components structured fairly well we can circle back around and copy the
components to schematics. There's a bit of friction regenerating the
platform from schematic each time.
The first thing most platforms need to do is come up with a strategy for
managing namespaces across multiple clusters.
This patch defines #Namespaces in the holos platform and adds a
namespaces component which loops over all values in the #Namespaces
struct and manages a kubernetes Namespace object.
The platform resource itself loops over all clusters in the platform to
manage all namespaces across all clusters.
From a blank slate:
```
❯ holos generate platform holos
4:26PM INF platform.go:79 wrote platform.metadata.json version=0.82.0 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=/home/jeff/workspace/holos-run/holos-infra/saas/platform.metadata.json
4:26PM INF platform.go:91 generated platform holos version=0.82.0 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=/home/jeff/workspace/holos-run/holos-infra/saas
❯ holos pull platform config .
4:26PM INF pull.go:64 pulled platform model version=0.82.0 server=https://jeff.app.dev.k2.holos.run:443 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc
4:26PM INF pull.go:75 saved platform config version=0.82.0 server=https://jeff.app.dev.k2.holos.run:443 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=platform.config.json
❯ (cd components && holos generate component cue namespaces)
4:26PM INF component.go:147 generated component version=0.82.0 name=namespaces path=/home/jeff/workspace/holos-run/holos-infra/saas/components/namespaces
❯ holos render platform ./platform/
4:26PM INF platform.go:29 ok render component version=0.82.0 path=components/namespaces cluster=management num=1 total=2 duration=464.055541ms
4:26PM INF platform.go:29 ok render component version=0.82.0 path=components/namespaces cluster=aws1 num=2 total=2 duration=467.978499ms
```
The result:
```sh
cat deploy/clusters/management/components/namespaces/namespaces.gen.yaml
```
```yaml
---
metadata:
name: holos
labels:
kubernetes.io/metadata.name: holos
kind: Namespace
apiVersion: v1
```
Without this patch the
holos.platform.v1alpha1.PlatformService.CreatePlatform doesn't work as
expected. The Platform message is used which incorrectly requires a
client supplied id which is ignored by the server.
This patch allows the creation of a new platform by reusing the update
operation as a mutation that applies to both create and update. Only
modifiable fields are part of the PlatformMutation message.
This patch adds a command to generate CUE based holos components from
examples embedded in the executable. The examples are passed through
the go template rendering engine with values pulled from flags.
Each directory in the embedded filesystem becomes a unique command for
nice tab completion. The `--name` flag defaults to "example" and is the
resulting component name.
A follow up patch with more flags will set the stage for a Helm
component schematic.
```
holos generate component cue minimal
```
```txt
3:07PM INF component.go:91 generated component version=0.80.2 name=example path=/home/jeff/holos/dev/bare/components/example
```
Split holos render into component and platform.
This patch splits the previous `holos render` command into subcommands.
`holos render component ./path/to/component/` behaves as the previous
`holos render` command and renders an individual component.
The new `holos render platform ./path/to/platform/` subcommand makes
space to render the entire platform using the platform model pulled from
the PlatformService.
Starting with an empty directory:
```sh
holos register user
holos generate platform bare
holos pull platform config .
holos render platform ./platform/
```
```txt
10:01AM INF platform.go:29 ok render component version=0.80.2 path=components/configmap cluster=k1 num=1 total=1 duration=448.133038ms
```
The bare platform has a single component which refers to the platform
model pulled from the PlatformService:
```sh
cat deploy/clusters/mycluster/components/platform-configmap/platform-configmap.gen.yaml
```
```yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
name: platform
namespace: default
data:
platform: |
spec:
model:
cloud:
providers:
- cloudflare
cloudflare:
email: platform@openinfrastructure.co
org:
displayName: Open Infrastructure Services
name: ois
```
This patch adds a subcommand to pull the data necessary to construct a
PlatformConfig DTO. The PlatformConfig message contains all of the
fields and values necessary to build a platform and the platform
components. This is an alternative to holos passing multiple tags to
CUE. The PlatformConfig is marshalled and passed once.
The platform config is also stored in the local filesystem in the root
directory of the platform. This enables repeated local building and
rendering without making an rpc call.
The build / render pipeline is expected to cache the PlatformConfig once
at the start of the pipeline using the pull subcommand.
We need to output a kind: Platform resource from cue so holos can
iterate over each build plan. The platform resource itself should also
contain a copy of the platform model obtained from the PlatformService
so holos can easily pass the model to each BuildPlan it needs to execute
to render the full platform.
This patch lays the groundwork for the Platform resource. A future
patch will have the holos cli obtain the platform model and inject it as
a JSON encoded string to CUE. CUE will return the Platform resource
which is a list of references to build plans. Holos will then iterate
over each build plan, pass the model back in, and execute the build
plan.
To illustrate where we're headed, the `cue export` step will move into
`holos` with a future patch.
```
❯ holos register user
3:34PM INF register.go:77 user version=0.80.0 email=jeff@ois.run server=https://app.dev.k2.holos.run:443 user_id=018f8839-3d74-7e39-afe9-181ad2fc8abe org_id=018f8839-3d74-7e3a-918c-b36494da0115
❯ holos generate platform bare
3:34PM INF generate.go:79 wrote platform.metadata.json version=0.80.0 platform_id=018f8839-3d74-7e3b-8cb8-77a2c124d173 path=/home/jeff/holos/dev/bare/platform.metadata.json
3:34PM INF generate.go:91 generated platform bare version=0.80.0 platform_id=018f8839-3d74-7e3b-8cb8-77a2c124d173 path=/home/jeff/holos/dev/bare
❯ holos push platform form .
3:34PM INF push.go:70 pushed: https://app.dev.k2.holos.run:443/ui/platform/018f8839-3d74-7e3b-8cb8-77a2c124d173 version=0.80.0
❯ cue export ./platform/
{
"metadata": {
"name": "bare",
"labels": {},
"annotations": {}
},
"spec": {
"model": {}
},
"kind": "Platform",
"apiVersion": "holos.run/v1alpha1"
}
```
This sub-command renders the web app form from CUE code and updates the
form using the `holos.platform.v1alpha1.PlatformService/UpdatePlatform`
rpc method.
Example use case, starting fresh:
```
rm -rf ~/holos
mkdir ~/holos
cd ~/holos
```
Step 1: Login
```sh
holos login
```
```txt
9:53AM INF login.go:40 logged in as jeff@ois.run version=0.79.0 name="Jeff McCune" exp="2024-05-17 21:16:07 -0700 PDT" email=jeff@ois.run
```
Step 2: Register to create server side resources.
```sh
holos register user
```
```
9:52AM INF register.go:68 user version=0.79.0 email=jeff@ois.run user_id=018f826d-85a8-751d-81ee-64d0f2775b3f org_id=018f826d-85a8-751e-98dd-a6cddd9dd8f0
```
Step 3: Generate the bare platform in the local filesystem.
```sh
holos generate platform bare
```
```txt
9:52AM INF generate.go:79 wrote platform.metadata.json version=0.79.0 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 path=/home/jeff/holos/platform.metadata.json
9:52AM INF generate.go:91 generated platform bare version=0.79.0 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 path=/home/jeff/holos
```
Step 4: Push the platform form to the `holos server` web app.
```sh
holos push platform form .
```
```txt
9:52AM INF client.go:67 updated platform version=0.79.0 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 duration=73.62995ms
```
At this point the platform form is published and functions as expected
when visiting the platform web interface.
When the user generates a platform, we need to know the platform ID it's
linked to in the holos server. If there is no platform with the same
name, the `holos generate platform` command should error out.
This is necessary because the first thing we want to show is pushing an
updated form to `holos server`. To update the web ui the CLI needs to
know the platform ID to update.
This patch modifies the generate command to obtain a list of platforms
for the org and verify the generated name matches one of the platforms
that already exists.
A future patch could have the `generate platform` command call the
`holos.platform.v1alpha1.PlatformService.CreatePlatform` method if the
platform isn't found.
Results:
```sh
holos generate platform bare
```
```txt
4:15PM INF generate.go:77 wrote platform.metadata.json version=0.77.1 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 path=/home/jeff/holos/platform.metadata.json
4:15PM INF generate.go:89 generated platform bare version=0.77.1 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 path=/home/jeff/holos
```
```sh
cat platform.metadata.json
```
```json
{
"id": "018f826d-85a8-751f-96d0-0d2bf70df909",
"name": "bare",
"display_name": "Bare Platform"
}
```
This patch adds a `holos register user` command. Given an authenticated
id token and no other record of the user in the database, the cli tool
use the API to:
1. User is registered in `holos server`
2. User is linked to one Holos Organization.
3. Holos Organization has the `bare` platform.
4. Holos Organization has the `reference` platform.
5. Ensure `~/.holos/client-context.json` contains the user id and an
org id.
The `holos.ClientContext` struct is intended as a light weight way to
save and load the current organization id to the file system for further
API calls.
The assumption is most users will have only one single org. We can add
a more complicated config context system like kubectl uses if and when
we need it.
