Compare commits

...

378 Commits

Author SHA1 Message Date
Jeff McCune
4127804092 quickstart: v0.93.2 with schema.#Platform
Make sure go install works from the quickstart documentation by doing a
release.  Otherwise, v0.93.1 is installed which doesn't include the
platform schema.
2024-09-09 17:04:32 -07:00
Jeff McCune
8f424cfabe quickstart: sync docs to this commit
Sync the documentation to the current output of the code at this commit.
2024-09-09 17:02:53 -07:00
Jeff McCune
699148abdd quickstart: define a convenince schema for the Platform
Previously, the quickstart step of generating the pod info component and
generating the platform as a whole left the task of integrating the
Component into the Platform as an exercise for the reader.  This is a
problem because it creates unnecessary friction.

This patch addresses the problem by lifting up the Platform concept
into the user-facing Schema API.  The generated platform includes a top
level #Platform definition which exposes the core Platform specification
on the Output field.

The Platform CUE instance then reduces to a simple `#Platform.Output`
which provides the Platform spec to holos for rendering each component
for each cluster.

The CUE code for the schema.#Platform iterates over each
Component to derive the list of components to manage for the Platform.

The CUE code for the generated quickstart platform links the definition
of StandardFleets, which is a Workload fleet and a Management cluster
fleet to the Platform conveninece wrapper.

Finally, the generated podinfo component drops a CUE file at the
repository root to automatically add the component to every workload
cluster.

The result is the only task left for the end user is to define at least
one workload cluster.  Once defined, the component is automatically
managed because it is managed on all workload clusters.

This approach futher opens the door to allow generated components to
define their namespaces and generated secrets on the management cluster
separate from their workloads on the workload clusters.

This patch includes a behavior change, from now on all generated
components should assume they are writing to the root of the user's Git
repository so that they can generate files through the whole tree.

In the future, we should template output paths for generated components.
A simple approach might be to embed a file with a .target suffix, with
the contents being a simple Go template of the file path to write to.
The holos generate subcommand can then check if any given embedded file
foo has a foo.target companion, then write the target to the rendered
template value.
2024-09-09 16:05:00 -07:00
Jeff McCune
73f777759e quickstart: mix-in argocd application resource
Users need to customize the default behavior of the core components,
like the Helm schema wrapper to mix-in an ArgoCD Application resource to
each component.  This patch wires up #Helm in the holos package to
schema.#Helm from the v1alpha3 api.

The result is illustrated in the Quickstart documentation, it is now
simple for users to modify the definition of a Helm component such that
Application resources are mixed in to every component in the platform.
2024-09-09 14:09:24 -07:00
Jeff McCune
8b9070f185 api: add schema to platform cue.mod for consistency
Previosly the end user needed to write, or at least copy and paste, a
large amount of boiler plate code to achieve the goal of declaring a
helm chart component.  There is a gap between the cue code:

    (#Helm & Chart).Output

And the full BuildPlan produced for the Holos cli to execute the
rendering process.  The boiler plate code in schema.cue at the root of
the platform infrastructure repository was largely responsible for
defining how a BuildPlan with one HelmChart component is derived from
this #Helm definition.

This patch moves the definitions into a new, documented API named
`schema`.  End users are expected to define their own #Helm definition
using the schema.#Helm, like so in the root level schema.cue:

    #Helm: schema.#Helm
2024-09-09 11:22:36 -07:00
Jeff McCune
1e8861c8b7 builder: relax api version requirement to fix deploy-dev
Without this patch deployments to the dev environment are failing with
the following error when commits are pushed to the main branch.

    GIT_DETAIL=v0.93.0-3-g4db3fb4 GIT_SUFFIX= bash ./hack/deploy-dev
    Cloning into 'holos-infra'...
    could not validate
    could not run: could not validate invalid BuildPlan: apiVersion invalid: want: v1alpha3 have: v1alpha2 at internal/builder/builder.go:308
    could not run: could not render component: exit status 1 at internal/render/platform.go:48
    make: *** [Makefile:147: dev-deploy] Error 1

This patch removes the api version check in the build plan validation
function.  In the future, we should pass an interface internally in the
holos executable.

The result is holos render platform ./platform succeeds with this patch
applied.
2024-09-06 20:58:56 -07:00
Jeff McCune
bdc182f4eb quickstart: generate podinfo helm chart 2024-09-06 20:57:35 -07:00
Jeff McCune
4db3fb4ead api: optional platform.spec.model
Previously the CUE code needed to specify the Platform.spec.model field,
which created friction.  This patch adds a cue struct tag to unify the
field with an open struct.

    ❯ holos render platform ./platform --log-level=debug
    could not run: could not marshal cue instance platform: cue: marshal error: spec.model: cannot convert incomplete value "_" to JSON at internal/builder/platform.go:45
    spec.model: cannot convert incomplete value "_" to JSON

The render command completes successfully with this patch without the
user having to provide a value for the spec.model field.
2024-09-06 13:38:48 -07:00
Jeff McCune
1911c7fe01 generate: add bare bones quickstart platform
This patch adds the minimal amount of CUE code necessary to successfully
run the following two commands from the quickstart.

    holos generate platform quickstart
    holos render platform ./platform

The result is no componets are rendered, so nothing is done, but it does
succeeed.

This patch surfaces some friction and inconsistency with how the Model
is passed in and the initial structure of the _PlatformConfig.  The tags
are required otherwise holos errors out.
2024-09-06 12:16:59 -07:00
Jeff McCune
5e582ec5c6 generate: do not require registration when generating a platform
Without this patch the `holos generate platform` command automatically
makes an rpc call to holos server.  This creates friction for the
quickstart guide because we don't need to require users to register and
have an organization and platform already created in the server just to
generate a simple platform to exercise a simple helm chart component.

A future patch should implement the behavior of linking a server side
platform to a local git repository by making the API call to get the
platform ID then updating the platform.metadata.json file.
2024-09-06 11:27:05 -07:00
Jeff McCune
e3c3ab6799 api: establish core v1alpha3 for quickstart
Switch holos to use v1alpha3 so we can establish more of the CUE
structures in the documented API using Go structs.
2024-09-06 10:59:45 -07:00
Jeff McCune
f3a1aeaf3f website: tweak landing page features
Still not 100% satisfied with these.  We may want to focus on the high
level core values of Safe, Easy, and Consistent instead.
2024-09-06 08:36:46 -07:00
Jeff McCune
1be7d5597b website: fix sidebars to focus on the tooling 2024-09-05 15:43:20 -07:00
Jeff McCune
2dc492dba8 website: add component to the concepts page 2024-09-05 15:07:16 -07:00
Jeff McCune
1364467853 ci: fix linter 2024-09-04 14:35:56 -07:00
Jeff McCune
7f37ac6721 website: focus landing page on package management
Previously the landing page focused on Holos as a reference platform.
We're refocusing the release on the holos package management tool.  This
patch updates the landing page and adds placeholders for a new quick
start guide which will focus on wrapping a helm chart and a concepts
page which will provide a high level overview of how holos is unique
from other tools.
2024-09-04 13:35:18 -07:00
Jeff McCune
3f3a3e5bb0 website: upgrade docusaurus to 3.5.2
npm i @docusaurus/core@latest @docusaurus/plugin-client-redirects@latest \
    @docusaurus/preset-classic@latest @docusaurus/theme-mermaid@latest \
    @docusaurus/module-type-aliases@latest @docusaurus/tsconfig@latest \
    @docusaurus/types@latest
2024-09-04 09:19:48 -07:00
Jeff McCune
4dc923f540 workflow: fix make lint 2024-08-28 12:42:26 -07:00
Jeff McCune
963ca0e6a7 workflows: move to ubuntu-latest
The gha-rs private runner scale set is no longer necessary now that the
repository is public.
2024-08-28 09:33:15 -07:00
Jeff McCune
ce875e6c18 Revert "docs: KubeStart readme"
This reverts commit ef016948b7.
2024-08-28 09:14:53 -07:00
Jeff McCune
ef016948b7 docs: KubeStart readme 2024-08-26 15:09:21 -07:00
Jeff McCune
df65f103e6 try-holos: embed helm charts
In an effort to increase reliability when trying holos locally.  The
idea being generate to render platform should ideally work without a
network connection provided the executable has already been downloaded.

For example, to give a quick demo without a network connection.
2024-08-23 08:09:37 -07:00
Jeff McCune
98d9831167 try-holos: embed argocd install
Without this patch the argo install manifest may fail because the
resources are fetched from github.

This patch embeds the same resources to increase speed and reliability.
2024-08-23 08:04:33 -07:00
Jeff McCune
fcb0f7d27a try-holos: embed argocd crds
Without this patch the argo crds component takes a few seconds to render
and may fail because the resources are fetched from github.

This patch embeds the same resources to increase speed and reliability.
2024-08-23 07:59:13 -07:00
Jeff McCune
5f3c6a1cc4 try-holos: embed gateway api resources
Without this patch the gateway api component takes a few seconds to
render and may fail because the resources are fetched from github.

This patch embeds the same resources to increase speed and reliability.

Result:

    rendered components/gateway-api for cluster workload in 257.206208ms
2024-08-23 07:55:45 -07:00
Jeff McCune
3ab6ccd864 try-holos: clarify local-ca must be run every time
Building the cluster today I got hung up on a `ERR_CONNECTION_CLOSED`
error from Chrome when trying to access httpbin.

The problem was I forgot to run the local-ca script, thinking I already
had a local ca.  The problem is the script also copies the private key
to the cluster, so it must be run every time the cluster is created.

This patch clarifies the sequence.  When resetting, everything following
the Create the Cluster step needs to be executed.
2024-08-19 16:00:29 -07:00
Jeff McCune
fe168a1a3f try-holos: clarify authentication must come before userinfo
This tripped me up.
2024-08-08 08:51:55 -07:00
Nate McCurdy
4c0d0dd18b readme: Fix typos and md formatting 2024-07-31 14:48:50 -07:00
Jeff McCune
9d0a0b1ed5 workflows: deploy dev-holos-app after image publish (#228)
Previously the image is build on merge to main, but not deployed
anywhere.  This patch adds steps to the publish workflow to deploy the
image that was published using gitops and argocd.
2024-07-30 12:12:32 -07:00
Jeff McCune
b6c6e9bc2f readme: restore from holos generate platform k3d mistake 2024-07-30 10:03:25 -07:00
Jeff McCune
44b560194a publish: add gha workflow to publish images with ko (#225)
Closes: #225
2024-07-29 17:17:32 -07:00
Jeff McCune
b545df9641 try-holos: tweak platform model section 2024-07-29 16:19:55 -07:00
Jeff McCune
e335541c6c make: fix latest connect tools installed
On a release, make tools is run which pulls in the latest connect tools
for angular.  This is a problem because it makes the git tree dirty.

The packages should be in the package.json file and the lock file so
these additional steps should not be necessary.

Remove them.

Desired result is make tools is idempotent and installs the correct
pinned versions necessary to build and release the container image.
2024-07-29 15:14:33 -07:00
Jeff McCune
3c1fcd9d6e cli: remove unused subcommands (#223)
This patch cleans up the cli commands, improves the short, long, and use
help strings, and makes some other minor changes for publishing the
code.
2024-07-29 14:41:59 -07:00
Jeff McCune
4fca94d863 doc: consolidate docs into doc (#223)
Examples are no longer needed, the current place for them is
the internal/generate/platform package.
2024-07-29 13:18:31 -07:00
Jeff McCune
a3d49f0d6e try-holos: incorporate nates edits
Incorporate most but not all of Nate's edits.
2024-07-26 14:07:49 -07:00
Nate McCurdy
f432a445a0 Edits to the getting stated guide after another run through
This makes the following changes to the getting started guide after
running through both the signed-in and signed-out paths:

* Added helm and git as requirements
* made it easier to modify the requirements by using all "1." list items
* Wait for the httpbin pod to be ready before continuing
* Make all the signed-out steps work
* Fixed sub-section header values so they show up in the TOC
* Fix minor typos and grammar issues
* Fix minor spacing and formatting inconsistencies
* Mark the ArgoCD guide as "coming soon"

Also fixed the docs for running the website locally to be able to
preview all these changes while working on them.
2024-07-25 12:34:01 -07:00
Jeff McCune
effaa9badf glossary: initial draft by gpt4o (#218)
GPT-4o got the initial definitions close enough for now, we'll refine
them as the reference platform continues to develop.
2024-07-24 13:13:40 -07:00
Jeff McCune
ac6be04859 try-holos: clarify rbac section (#218)
It wasn't clear to Nate what this section was for because it was
awkwardly placed after the heavy edits recently.
2024-07-24 12:41:30 -07:00
Jeff McCune
c0ca7e7392 try-holos: another run-through (#218)
One more run through of Try Holos Locally from top to bottom.
2024-07-24 09:48:48 -07:00
Jeff McCune
2f0b883724 try-holos: another run-through (#218)
One more run through of Try Holos Locally from top to bottom.
2024-07-24 09:31:28 -07:00
Jeff McCune
7b8eed0347 try-holos: redirect /docs/tutorial/local/k3d (#218)
Redirect /docs/tutorial/local/k3d/ to /docs/guides/try-holos/

Cloudflare is still serving up the old page even though it's no longer
being built.
2024-07-24 07:37:36 -07:00
Jeff McCune
230a2f18b8 try-holos: button up try holos locally (#218)
Noticed a few remaining rough edges when I read through it on my phone
last night.  This patch hopefully gets the try holos doc into a place
we're happy with.
2024-07-24 07:25:47 -07:00
Jeff McCune
89578d891f try-holos: organize into guides (#218)
Instead of tutorials.  The goal is to refine Try Holos Locally down to a
minimal number of steps and then branch out to deeper use cases like
ArgoCD, Backstage, etc...

This patch moves the ArgoCD related sections to a separate "dive deeper"
guide to trim down the length of the try holos guide.
2024-07-23 21:35:47 -07:00
Jeff McCune
8995af06fa local-k3d: enable server side apply auto sync (#218)
The postgres crds exceed 256Ki and need server side apply.
2024-07-23 16:44:03 -07:00
Jeff McCune
55752aee1c local-k3d: enable anonymous access to argocd (#216)
When someone is trying holos locally but has not signed up, ArgoCD needs
to be configured to allow anonymous access.  This patch enables
anonymous access and gives the admin role.

With this patch the Try Holos Locally guide can be completed without
signing up or signing in.
2024-07-23 13:48:18 -07:00
Jeff McCune
a90ba17904 local-k3d: try holos without sign-up (#216)
Enable people to try holos without having to sign up at all.  This is
through the ArgoCD section.
2024-07-23 12:15:06 -07:00
Jeff McCune
6f78984561 local-k3d: add clean up section (#216)
It's nice to know how to clean up before starting toward the goal, it
sets a boundary.
2024-07-23 06:21:11 -07:00
Jeff McCune
b927caed96 quickstart: accept *.local domains for Orb (#200)
Nate gave the feedback the Try Holos Locally doesn't work with Orb.
This patch makes the input form accept *.local domains so we can use the
default Orb managed domain of *.k8s.orb.local

I haven't tested this, but we at least need to allow the domain to
test it.

[1]: https://docs.orbstack.dev/kubernetes/#loadbalancer-ingress
2024-07-23 05:59:28 -07:00
dependabot[bot]
e4e8a5e217 build(deps): bump ws, engine.io and socket.io-adapter
Bumps [ws](https://github.com/websockets/ws), [engine.io](https://github.com/socketio/engine.io) and [socket.io-adapter](https://github.com/socketio/socket.io-adapter). These dependencies needed to be updated together.

Updates `ws` from 8.17.0 to 8.17.1
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](https://github.com/websockets/ws/compare/8.17.0...8.17.1)

Updates `engine.io` from 6.5.4 to 6.5.5
- [Release notes](https://github.com/socketio/engine.io/releases)
- [Changelog](https://github.com/socketio/engine.io/blob/6.5.5/CHANGELOG.md)
- [Commits](https://github.com/socketio/engine.io/compare/6.5.4...6.5.5)

Updates `socket.io-adapter` from 2.5.4 to 2.5.5
- [Release notes](https://github.com/socketio/socket.io-adapter/releases)
- [Changelog](https://github.com/socketio/socket.io-adapter/blob/2.5.5/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io-adapter/compare/2.5.4...2.5.5)

---
updated-dependencies:
- dependency-name: ws
  dependency-type: indirect
- dependency-name: engine.io
  dependency-type: indirect
- dependency-name: socket.io-adapter
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-22 22:39:42 +00:00
Jeff McCune
804bafd4e6 security: fix RCE on git-go clients
Closes: #214
2024-07-22 15:37:00 -07:00
Jeff McCune
f2a9508aba try holos: additional tweaks to try holos locally 2024-07-22 15:33:04 -07:00
Jeff McCune
392b9f711b logging: make top level logger console not json
Previously the top level logger used a json handler while the rest of
the code used the default console handler.  This patch unifies them to
be consistent.
2024-07-22 15:03:11 -07:00
Jeff McCune
2d9f35067f tutorial: update try holos locally
Remove side comments about the reference platform.  Move the in-line
exploration of ArgoCD and CUE to the end once the reader has completed
their goal.  Other minor edits.
2024-07-22 11:56:01 -07:00
Jeff McCune
a0fd53deaa builder: fix cue panic (#212)
Previously CUE paniced when holos tried to unify values originating from
two different cue runtimes.  This patch fixes the problem by
initializaing cue.Value structs from the same cue context.

Log messages are also improved after making one complete pass through
the Try Holos Locally guide.
2024-07-22 10:14:32 -07:00
Jeff McCune
e346e10c07 v0.91.0 2024-07-21 21:23:48 -07:00
Jeff McCune
f1dc54650e builder: fill #UserData from userdata/**/*.json (#210)
Now that we have multi-platform images, we need a way to easily deploy
them.  This involves changing the image tag.  kustomize edit is often
used to bump image tags, but we can do better providing it directly in
the unified CUE configuration.

This patch modifies the builder to unify user data *.json files
recursively under userdata/ into the #UserData definition of the holos
entrypoint.

This is to support automation that writes simple json files to version
control, executes holos render platform, then commits and pushes the
results for git ops to take over deployment.

The make deploy target is the reason this change exists, to demonstrate
how to automatically deploy a new container image.
2024-07-21 21:22:22 -07:00
Jeff McCune
9ed5d588d0 makefile: make image for Multi-Platform Images (#209)
Use ko to build a multi-platform image.

Closes: #209
2024-07-21 20:12:09 -07:00
Nate McCurdy
6eb24faf63 cli/delete: improve platform deletion help text and output (#200)
- Clarify help text to indicate one or more platform IDs as arguments.
- Show platform name and ID in `delete platform` output for clarity.
2024-07-21 09:55:12 -07:00
Jeff McCune
daa13906b5 add make tag target 2024-07-21 09:33:24 -07:00
Jeff McCune
b2ce455aa2 generate/platform: use platform metadata as source of truth (#200)
This patch addresses Nate's feedback that it's difficult to know what
platform is being operated on.

Previously it wasn't clear where the platform id used for push and pull
comes from.  The source of truth is the platform.metadata.json file
created when the platform is first generated using `holos generate
platform k3d`.

This patch removes the platformId field from the platform.config.json
file, renames the platform.config.json file to platform.model.json and
renames the internal symbols to match the domain language of "Platform
Model" instead of the less clear "config"

This patch also changes the API between holos and CUE to use the proto
json imported from the proto file instead of generated from the go code
generated from the proto file.  The purpose is to ensure protojson
encoding is used end to end.

Default log handler:

The patch also changes the default log output to print only the message
to stderr.  This addresses similar feedback from both Gary and Nate that
the output is skipped over because it feels like internal debug logs.

We still want 100% of output to go through the logger so we can ensure
each line can be made into valid json.  Info messages however are meant
for the user and all other attributes can be stripped off by default.
If additional source location is necessary, enable the text or json
output format.

Protobuf JSON:

This patch modifies the API contract between holos and CUE to ensure
data is exchanged exclusively using protojson.  This is necessary
because protobuf has a canonical json format which is not compatible
with the go json package struct tags.  When Holos handles a protobuf
message, it must marshal and unmarshal it using the protojson package.

Similarly, when importing protobuf messages into CUE, we must use `cue
import` instead of `cue go get` so that the canonical format is used
instead of the invalid go json struct tags.

Finally, when a Go struct like v1alpha1.Form is used to represent data
defined in cue which contains a nested protobuf message, Holos should
use a cue.Value to lookup the nested path, marshal it into json bytes,
then unmarshal it again using protojson.
2024-07-21 09:31:09 -07:00
Jeff McCune
90629b84b5 cli/delete: delete platform by uuid (#200)
Previously there was no way to delete a platform.  This patch adds a
basic delete subcommand which deletes platforms by their id using the
rpc api.

    ❯ holos get platform
    NAME         DESCRIPTION        AGE    ID
    k3d          Holos Local k3d    20h    0190c78a-4027-7a7e-82d0-0b9f400f4bc9
    k3d2         Holos Local k3d    20h    0190c7b3-382b-7212-81d6-ffcfc4a3fe7e
    k3dasdf      Holos Local k3d    20h    0190c7b3-728a-7212-b56d-2d2edf389003
    k3d9         Holos Local k3d    20h    0190c7b8-4c4e-7cea-9d3d-a6b9434ae438
    k3d-8581     Holos Local k3d    20h    0190c7ba-1de9-7cea-bff8-f15b51a56bdd
    k3d-13974    Holos Local k3d    20h    0190c7ba-5833-7cea-b863-8e5ffb926810
    k3d-20760    Holos Local k3d    19h    0190c7ba-7a12-7cea-a350-d55b4817d8bc

    ❯ holos delete platform 0190c7ba-1de9-7cea-bff8-f15b51a56bdd 0190c7ba-5833-7cea-b863-8e5ffb926810 0190c7ba-7a12-7cea-a350-d55b4817d8bc
    deleted platform k3d-8581
    deleted platform k3d-13974
    deleted platform k3d-20760
2024-07-19 09:54:48 -07:00
Jeff McCune
7f077f5347 cli/get: list platforms (#200)
Previously there was no way to get/list platforms.  This patch adds a
basic get subcommand with list as an alias to get the platforms
currently defined in the organization.

    ❯ holos get platform
    NAME         DESCRIPTION        AGE    ID
    k3d          Holos Local k3d    18h    0190c78a-4027-7a7e-82d0-0b9f400f4bc9
    k3d2         Holos Local k3d    17h    0190c7b3-382b-7212-81d6-ffcfc4a3fe7e
    k3dasdf      Holos Local k3d    17h    0190c7b3-728a-7212-b56d-2d2edf389003
    k3d9         Holos Local k3d    17h    0190c7b8-4c4e-7cea-9d3d-a6b9434ae438
    k3d-8581     Holos Local k3d    17h    0190c7ba-1de9-7cea-bff8-f15b51a56bdd
    k3d-13974    Holos Local k3d    17h    0190c7ba-5833-7cea-b863-8e5ffb926810
    k3d-20760    Holos Local k3d    17h    0190c7ba-7a12-7cea-a350-d55b4817d8bc
    k3d-13916    Holos Local k3d    17h    0190c7ba-8313-7cea-be37-41491c95ae79
    k3d-26154    Holos Local k3d    17h    0190c7ba-a117-7cea-8229-ce27da84135e

    ❯ holos get platform foo
    7:16AM ERR could not execute version=0.89.1 code=unknown err="not found"

    ❯ holos get platform foo k3d
    NAME    DESCRIPTION        AGE    ID
    k3d     Holos Local k3d    18h    0190c78a-4027-7a7e-82d0-0b9f400f4bc9
2024-07-19 07:16:43 -07:00
Jeff McCune
327193215b version: 0.89.1 2024-07-18 14:42:24 -07:00
Jeff McCune
b98b5cae3f PlatformService: do nothing when platform already exists
Previously the CreatePlatform rpc wrote over all fields when the
platform already exists.  This is surprising and basically the
UpdatePlatform rpc.

This patch changes the behavior to do nothing except set the
already_exists flag in the response message.

Users who have the use case of needing to know if the creation actually
created a new resource should use the API to check the already_exists
flag.  The CLI has no affordance for this other than parsing the log
messages.
2024-07-18 14:27:03 -07:00
Jeff McCune
ddba69517f version 0.89.0 2024-07-18 12:00:49 -07:00
Jeff McCune
e28642bca7 platform service: make (#205) create platform should be idempotent
Previously holos.platform.v1alpha1.PlatformService.CreatePlatform
returns an error for a request to create a platform of the same name as
an existing platform.

    holos create platform --name k3d --display-name "Try Holos Locally"

    8:00AM ERR could not execute version=0.87.2 code=failed_precondition
    err="failed_precondition: platform.go:55: ent: constraint failed:
    ERROR: duplicate key value violates unique constraint
    \"platform_org_id_name\" (SQLSTATE 23505)" loc=client.go:138

This patch makes the CreatePlatform rpc idempotent using the upsert API.
The already_exists bool field is added to CreatePlatformResponse
response to indicate to the client if the platform already exists or
not.

Result:

    holos create platform --display-name "Holos Local" --name k3d10

    11:53AM INF create.go:56 created platform k3d10 version=0.87.2
    name=k3d10 id=0190c731-1808-7e7d-9ccb-3d17434d0055
    org=0190c6d6-4974-7733-9f7b-5d759a3e60e7 exists=false

    holos create platform --display-name "Holos Local" --name k3d10

    11:53AM INF create.go:56 updated platform k3d10 version=0.87.2
    name=k3d10 id=0190c731-1808-7e7d-9ccb-3d17434d0055
    org=0190c6d6-4974-7733-9f7b-5d759a3e60e7 exists=true
2024-07-18 11:53:51 -07:00
Jeff McCune
dceb37b7ab tilt: run holos server locally in k3d (#205)
Previously I developed holos server in the dev-holos namespace of a
remote cluster.  This patch updates the Tilt configs to develop locally
against k3d quickly and easily.

The database is a CNPG database which replaces PGO.  This is simpler and
ligher weight, one container in one pod.  CNPG has no repo host like PGO
has.
2024-07-18 10:24:45 -07:00
Jeff McCune
f6340ea4fe runbooks: recover zitadel with cnpg
Replaces the previous PGO runbook, we no longer use PGO and use CNPG
instead.
2024-07-18 07:58:58 -07:00
Jeff McCune
3845174738 server: add holos server init subcommand for migration (#204)
When starting holos server from the production Deployment, pgbouncer
blocks the automatic migration on startup.

```json
{
  "time": "2024-07-16T16:35:52.54507682-07:00",
  "level": "ERROR",
  "msg": "could not execute",
  "version": "0.87.2",
  "code": "unknown",
  "err": "sql/schema: create \"users\" table: ERROR: permission denied for schema public (SQLSTATE 42501)",
  "loc": "cli.go:82"
}
```

This patch separates automatic migration into a `holos server init`
subcommand intended for use in a Job.

Closes: #204
2024-07-16 17:55:40 -07:00
Jeff McCune
f0bc21a606 tilt: local development using k3d (#200)
Previously, the Tiltfile was hard-wired to Jeff's development
environment on the k2 cluster on-prem.  This doesn't work for other
contributors.

This patch fixes the problem by re-using the [Try Holos Locally][1]
documentation to create a local development enironment.  This has a
number of benefits.  The evaluation documentation will be kept up to
date because it doubles as our development environment.  Developing
locally is preferrable to developing in a remote cluster.  Hostnames and
URL's can be constant, e.g. https://app.holos.localhost/ for local dev
and https://app.holos.run/ for production.  We don't need to push to a
remote container registry, k3d has a local registry built in that works
with Tilt.

The only difference presently between evaluation and development when
following the local/k3d doc is the addition of a local registry.

With this patch holos starts up and is accessible at
https://app.holos.localhost/

[1]: https://holos.run/docs/tutorial/local/k3d/
2024-07-15 17:08:33 -07:00
Jeff McCune
6d0e48fccb github/workflows: disable test workflow
until we allocate time to fix it
2024-07-15 12:20:13 -07:00
Nate McCurdy
f5035ce699 docs/website: Touch up the k3d tutorial
This applies various grammar, formatting, and flow improvements to the
local k3d tutorial steps based on running through it from start to
finish.

This also removes the Go code responsible for embedding the website into
`holos`, which isn't needed since the site is hosted on Cloudflare
Pages.
2024-07-15 11:37:23 -07:00
Jeff McCune
3c694d2a92 doc/website: final first pass at local k3d (#199)
Link it off the nav, footer, and sidebar.  Follow up with another task
to reorganize and slim it down.

Closes: #199
2024-07-14 19:45:56 -07:00
Jeff McCune
b8592b0b72 doc/website: add holos social card (#199)
Made it in preview using a background png from https://social.cards/ and
converting our logo.

    mogrify -background none -resize 1200x -format png logo.svg
2024-07-14 14:38:17 -07:00
Jeff McCune
cf2289ef19 doc/website: make try holos next after intro (#199)
Previously the intro page linked next to the glossary.  This patch makes
the try holos locally page immediately follow the introduction page.
2024-07-14 14:06:55 -07:00
Jeff McCune
5e5b9c97d4 doc/website: fix link and mermaid colors (#199)
This patch fixes up the link colors and mermaid diagrams to look better
in both light and dark mode.  This may not be the final result but it
moves in the right direction.

Links are now blue with a visible line on hover.
2024-07-14 13:34:02 -07:00
Jeff McCune
a19e0ff3f3 doc/website: fix spelling errors (#199)
This patch adds cspell over doc/md to the make lint task and fixes
existing spelling errors in the documentation.
2024-07-14 12:48:31 -07:00
Jeff McCune
ac632cb407 doc/website: sync ArgoCD Applications automatically (#199)
Previously the guide did not cover reconciling holos platform components
with GitOps.  This patch adds instructions on how to apply the
application resources, review the diff, sync manually, and finally
enable automatic sync using CUE's struct merge feature.
2024-07-14 10:02:22 -07:00
Jeff McCune
154bbabf01 doc/website: add argocd to k3d platform (#199)
Previously there is no web app except httpbin in the k3d platform.  This
commit adds ArgoCD with an httproute and authorization policy at the
mesh layer.  The application layer authenticates against a separate
oidc client id in the same issuer the mesh uses to demonstrate zero
trust and compatibility between the application and platform layers.

With this patch the user can authenticate and log in, but applications
are not configured.  The user has no roles in ArgoCD either, rbac needs
to be configured properly for the getting started guide.
2024-07-14 06:56:15 -07:00
Jeff McCune
95e45d59cb doc/website: clarify why we use httpbin (#199)
Useful to inspect request headers from the perspective of the backend.
2024-07-13 19:50:26 -07:00
Jeff McCune
a45abedd32 doc/website: touch up process after a run through (#199)
Clean up, touch up.
2024-07-13 19:36:08 -07:00
Jeff McCune
a644b1181b doc/website: move rendering section to k3d (#199)
Previously the intro was spread out.  This patch focuses the tutorial
solely onto the k3d process.
2024-07-13 14:24:44 -07:00
Jeff McCune
861b552b0b doc/website: add k3d authproxy and authpolicy (#199)
This patch adds the authproxy and authpolicy holos components to the k3d
platform for local evaluation.  This combination implements a basic Zero
Trust security model.  The httpbin backend service is protected with
authenication and authorization at the platform level without any
changes to the backend service.

The client id and project are static because they're defined centrally
in https://login.holos.run to avoid needing to setup a full identity
provider locally in k3d.

With this patch authentication and authorization work from both the web
browser and from the command line with curl using the token provided by
the holos cli.
2024-07-13 14:09:41 -07:00
Jeff McCune
5d0212e832 doc/website: local k3d with httpbin working (#199)
Previously the local k3d tutorial doesn't expose any services to verify
the local certificate and the local dns changes work as expected.

This patch adds instructions and modifies the k3d platform to work with
a local mkcert certificate.  A ClusterIssuer is configured to issue
Certificate resources using the ca private key created my mkcert.

With this patch, following the instructions results in a working and
trusted httpbin resource at https://httpbin.holos.localhost  This works
both in Chrome and curl on the command line.
2024-07-13 07:35:44 -07:00
Jeff McCune
9f434928d6 doc/website: add istio gateway and local ca (#199)
This patch adds a script to install a local CA and configure cert
manager to issue certs similar to how it issues certs using LetsEncrypt
in a real cluster.
2024-07-12 10:19:30 -07:00
Jeff McCune
5b1fa4b046 doc/website: add helm chart cue example (#199)
This patch adds an example of how Holos uses unmodified upstream helm
charts to integrate software projects into a platform.
2024-07-11 21:27:29 -07:00
Jeff McCune
ae4614c35b internal/generate: add k3d platform and tutorial (#199)
Previously there is no way to evaluate Holos on local host.  This is a
problem because it's a high barrier to entry to setup a full blown GKE
and EKS cluster to evaluate the reference platform.

This patch adds a minimal, but useful, k3d platform which deploys to a
single local k3d cluster.  The purpose is to provide a shorter on ramp
to see the value of ArgoCD integrated with Istio to provide a zero trust
auth proxy.

The intentional trade off is to provide a less-holistic k3d platform
with a faster on-ramp to learn about the value the more-holistic holos
platform.

With this patch the documentation is correct and the platform renders
fully.  The user doesn't need to provide any Platform Model values, the
default suffice.

For the ArgoCD client ID, we'll use https://login.holos.run as the
issuer instead of building a new OIDC issuer inside of k3d, which would
create significant friction.
2024-07-11 21:07:05 -07:00
Jeff McCune
e99a00f0a1 doc/website: fix API reference docs links in header and footer
Previously the API nav link went to the CLI docs which was weird.
Should go to the current API reference docs.
2024-07-11 11:36:30 -07:00
Jeff McCune
e89dcb9783 doc/website: tagline: The Platform Operating System
Gary and I chatted about this yesterday.  Best tagline we've come up
with so far driving at the analogy with a debian distribution.
2024-07-11 10:44:13 -07:00
Jeff McCune
05806cb439 doc/website: add rendering pipeline diagram
This patch adds a diagram that gives an overview of the holos rendering
pipeline.  This is an importantn concept to understand when working with
holos components.

Note this probably should not go in the Overview, which is intended only
to give a sense of what getting started looks like.  Move it to the
render page when we add it.
2024-07-07 14:16:46 -07:00
Jeff McCune
bfb8458bcb doc/website: draft architecture
This patch fills out some of the architecture page.  Not totally happy
with it yet but it's a start.
2024-07-07 13:28:15 -07:00
Jeff McCune
55d4033116 doc/website: add mermaid architecture diagram
Previously there are no diagrams in the documentation.  This patch wires
up mermaid for use in code blocks in the markdown files.  A minimal
diagram is added to verify mermaid works but it's not the final diagram.
2024-07-07 08:54:22 -07:00
Jeff McCune
276dc95029 doc/website: tweak observability feature
Madison says it sounds weird.
2024-07-06 16:57:16 -07:00
Jeff McCune
c473321817 doc/website: add holos features on landing page
Previously the Docusaurus features examples were still in place on the
home page.  This patch replaces the homepage features with Holos
specific features and illustrations from undraw.

Refer to https://undraw.co/search
2024-07-06 16:11:07 -07:00
Jeff McCune
1b539b8874 cmd/holos: version 0.87.2 2024-07-06 11:15:28 -07:00
Jeff McCune
1892fe31ae doc/website: tweak the intro for readability 2024-07-06 09:27:30 -07:00
Jeff McCune
b9c1e49822 doc/website: remove holos website command (#198)
Generating the docusaurus site is not idempotent like generating the
Angular web app.  This is a problem for building and releasing the
executable because it creates a dirty git state.

Embedding the doc website into the executable is no longer necessary
since we're deploying the site with Cloudflare pages.  Remove it from
the compiled executable as a result.
2024-07-06 08:28:07 -07:00
Jeff McCune
f31a630139 doc/website: npm install in cloudflare (#198)
Cloudflare fails to build the website with:

```
07:44:47.179	sh: 1: docusaurus: not found
07:44:47.192	Failed: Error while executing user command. Exited with error code: 127
```

Resolve it by executing npm install from the build-website script and
note the script is intended for use in a cloudflare context.
2024-07-06 07:47:28 -07:00
Jeff McCune
a4445c7d17 doc/website: build and deploy to cloudflare pages (#198)
Previously the website isn't deployed.  Instead of building the
container and deploying it, deploy to cloudflare pages which has a
simple to use GitHub integration.

Refer to https://dev.to/gaurishhs/deploying-docusaurus-to-cloudflare-pages-565g
2024-07-06 07:42:55 -07:00
Jeff McCune
d0b392cfe0 docs/website: generate v1alpha2 api docs from source (#196)
The API docs are not published yet becuase the module is private.  Our
own docs site does not have any API reference docs.

This patch adds auto-generated markdown docs for the core v1alpha2 types
by generating them directly from the go source code.

Some light editing of the output of `gomarkdoc` is necessary to get the
heading anchor tags to align correctly for Docusaurus.
2024-07-04 14:51:30 -07:00
Jeff McCune
efc215dc8c doc/website: iterate on the intro page
Working on describing the features and value of Holos a bit more.
2024-07-04 11:04:40 -07:00
Jeff McCune
2f446bd60a doc/website: clean out blog template 2024-07-03 17:49:47 -07:00
Jeff McCune
92889cb9a4 doc/website: add landing page and basic docs (#88)
This patch adds the basics of a Holos landing page and initial markdown
docs.
2024-07-03 17:33:42 -07:00
Jeff McCune
c53e00f609 doc/website: fix workflows (#192)
The changes to the Makefile break github actions workflows.  This patch
fixes them.
2024-07-02 14:25:37 -07:00
Jeff McCune
5e2c0e7d64 doc/website: use npm instead of yarn (#192)
The github workflows fail because yarn is not available.  The Angular
frontend app uses npm so we should also use npm for the website to
minimize dependencies.
2024-07-02 14:11:52 -07:00
Jeff McCune
adbffe34d8 use go:generate and commit all results (#192)
Previously `go install` fails to install holos.

```
❯ go install github.com/holos-run/holos/cmd/holos@latest
../../go/pkg/mod/github.com/holos-run/holos@v0.86.0/internal/frontend/frontend.go:25:12: pattern holos/dist/holos/ui/index.html: no matching files found
../../go/pkg/mod/github.com/holos-run/holos@v0.86.0/doc/website/website.go:14:12: pattern all:build: no matching files found
```

This is because we do not commit required files.  This patch fixes the
problem by following Rob Pike's guidance to commit generated files.
This patch also replaces the previous use of Makefile tasks to generate
code with //go:generate directives.

This means the process of keeping the source code clean is straight
forward:

```
git clone
make tools
make generate
make build
```

Refer to https://go.dev/blog/generate

> Also, if the containing package is intended for import by go get, once
> the file is generated (and tested!) it must be checked into the source
> code repository to be available to clients. - Rob Pike
2024-07-02 13:50:11 -07:00
Jeff McCune
5e62008d78 doc/website: Add README (#84)
Previously there's no README to get starts.  This patch adds a README
modeled after ent's.
2024-07-02 08:57:20 -07:00
Jeff McCune
af1c009dad doc/website: add holos website command to serve docusaurus (#84)
Previously docs are not published.  This patch adds Docusaurus into the
doc/website directory which is also a Go package to embed the static
site into the executable.

Serve the site using http.Server with a h2c handler with the command:

    holos website --log-format=json --log-drop=source

The website subcommand is intended to be run from a container as a
Deployment.  For expedience, the website subcommand doesn't use the
signals package like the server subcommand does. Consider using it for
graceful Deployment restarts.

Refer to https://github.com/ent/ent/tree/master/doc/website
2024-07-01 22:10:28 -07:00
Jeff McCune
53cb9ba7fb (#189) Make the v1alpha2 API data only
Previously a couple of methods were defined on the Result struct.

This patch moves the methods to an internal wrapper struct to remove
them from the API documentation.

With this patch the API between holos and CUE is entirely a data API.
2024-06-30 17:19:35 -07:00
Jeff McCune
4cc139b372 (#189) v1alpha2 Reference Docs 2024-06-30 16:13:12 -07:00
Jeff McCune
8bc7804a9c Merge pull request #190 from holos-run/jeff/189-reference-docs
(#189) v1alpha2 API for reference docs
2024-06-30 15:07:46 -07:00
Jeff McCune
a39807a858 (#189) go mod tidy 2024-06-30 15:04:39 -07:00
Jeff McCune
5170650760 (#189) Remove yaml tags for v1alpha2.
Unnecessary, json tags are sufficient for both yaml and json.
2024-06-30 14:50:50 -07:00
Jeff McCune
1d81b3c3b4 (#189) Clarify documentation of v1alpha2
Focusing on the purpose of #APIObjects
2024-06-30 14:36:04 -07:00
Jeff McCune
33970dafe8 (#189) Version 0.85.0 v1alpha2 2024-06-30 10:27:48 -07:00
Jeff McCune
faa46c54d8 (#189) Do not write empty files with gitops results
Previosly, the holos component Results for each ArgoCD Application
resource managed as part of each BuildPlan results in an empty file
being written for the empty list of k8s api objects.

This patch fixes the problem by skipping writing the accumulated output
of API objects with the Result metadata.name starts with `gitops/`.

This is kind of a hack, but it works well enough for now.
2024-06-30 10:15:28 -07:00
Jeff McCune
42509a34cf (#189) Fix the gitops Application component name
Previously components appeared to be duplicated, it was not clear to the
user one build plan results in two components: one for the k8s yaml and
one for the gitops argocd Application resource.

```
❯ holos render component --cluster-name aws1 components/login/zitadel-server
9:27AM INF result.go:195 wrote deploy file version=0.84.1 path=deploy/clusters/aws1/gitops/zitadel-server.application.gen.yaml bytes=338
9:27AM INF render.go:92 rendered zitadel-server version=0.84.1 cluster=aws1 name=zitadel-server status=ok action=rendered
9:27AM INF render.go:92 rendered zitadel-server version=0.84.1 cluster=aws1 name=zitadel-server status=ok action=rendered
```

This patch prefixes the ArgoCD Application resource, which is
implemented as a separate HolosComponent in the same BuildPlan.  The
result is more clear about what is going on:

```
❯ holos render component --cluster-name aws1 components/login/zitadel-server
9:39AM INF result.go:195 wrote deploy file version=0.84.1 path=deploy/clusters/aws1/gitops/zitadel-server.application.gen.yaml bytes=338
9:39AM INF render.go:92 rendered gitops/zitadel-server version=0.84.1 cluster=aws1 name=gitops/zitadel-server status=ok action=rendered
9:39AM INF render.go:92 rendered zitadel-server version=0.84.1 cluster=aws1 name=zitadel-server status=ok action=rendered
```
2024-06-30 09:37:14 -07:00
Jeff McCune
ef369d4860 (#189) Format cue code with make fmt
Previously the internal cue code was not formatted properly.  This patch
updates `make fmt` to automatically format the embedded internal
platforms.
2024-06-30 09:35:25 -07:00
Jeff McCune
747ed3462a (#189) Fix Helm + Kustomize post renderer for v1alpha2
Previously the `login/zitadel-server` component failed to render with
the following error.  This is a result of the kustomize config fileds
moving down one level to the `kustomize` field in v1alpha2 relative to
`v1alpha`.

```
spec.components.helmChartList.0.kustomizeFiles: field not allowed:
    ./buildplan.cue:106:9
    ./buildplan.cue:106:27
    ./buildplan.cue:118:3
    ./buildplan.cue:124:4
    ./buildplan.cue:125:4
    ./buildplan.cue:126:5
    ./buildplan.cue:162:10
    ./buildplan.cue:165:37
    ./buildplan.cue:206:13
    ./components/login/zitadel-server/zitadel.cue:9:1
    ./components/login/zitadel-server/zitadel.cue:18:9
    ./components/login/zitadel-server/zitadel.cue:19:9
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:31:8
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:36:15
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:42:19
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:42:22
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:48:18
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/helm_go_gen.cue:12:13
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/helm_go_gen.cue:13:2
spec.components.helmChartList.0.resourcesFile: field not allowed:
    ./buildplan.cue:106:9
    ./buildplan.cue:106:27
    ./buildplan.cue:118:3
    ./buildplan.cue:122:4
    ./buildplan.cue:125:4
    ./buildplan.cue:125:43
    ./buildplan.cue:162:10
    ./buildplan.cue:165:37
    ./buildplan.cue:206:13
    ./components/login/zitadel-server/zitadel.cue:9:1
    ./components/login/zitadel-server/zitadel.cue:18:9
    ./components/login/zitadel-server/zitadel.cue:19:9
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:31:8
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:36:15
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:42:19
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:42:22
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:48:18
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/helm_go_gen.cue:12:13
    ./cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/helm_go_gen.cue:13:2
_PlatformConfig: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./buildplan.cue:232:21
    ./schema.cue:14:44
_PlatformConfig: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./components/login/login.cue:6:18
    ./schema.cue:14:44
_PlatformConfig: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./components/login/zitadel.cue:8:18
    ./schema.cue:14:44
_PlatformConfig: invalid interpolation: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./platform.cue:61:17
    ./platform.cue:60:19
    ./schema.cue:14:44
_PlatformConfig: invalid interpolation: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./platform.cue:62:17
    ./platform.cue:60:19
    ./schema.cue:14:44
_PlatformConfig: invalid interpolation: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./platform.cue:79:17
    ./platform.cue:78:19
    ./schema.cue:14:44
_PlatformConfig: invalid interpolation: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./platform.cue:80:17
    ./platform.cue:78:19
    ./schema.cue:14:44
_PlatformConfig: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./platform.cue:100:25
    ./schema.cue:14:44
_PlatformConfig: invalid interpolation: invalid interpolation: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./platform.cue:102:22
    ./platform.cue:100:25
    ./schema.cue:14:44
_PlatformConfig: error in call to encoding/json.Unmarshal: json: invalid JSON:
    ./schema.cue:14:44
```

With this patch the component renders without any further modification:

```
❯ holos render component --cluster-name aws1 components/login/zitadel-server
9:24AM INF result.go:195 wrote deploy file version=0.84.1 path=deploy/clusters/aws1/gitops/zitadel-server.application.gen.yaml bytes=338
9:24AM INF render.go:92 rendered zitadel-server version=0.84.1 cluster=aws1 name=zitadel-server status=ok action=rendered
9:24AM INF render.go:92 rendered zitadel-server version=0.84.1 cluster=aws1 name=zitadel-server status=ok action=rendered
```
2024-06-30 09:23:01 -07:00
Jeff McCune
1fb1798f60 (#189) Make HolosComponent Metadata Namespace optional
Previously a metadata.namespace value was required for all holos
components.  This is a problem because not all resources require a
namespace, for example producing the ArgoCD Application resource for
each build plan does not need a namespace defined, particularly when
managing only CRDs.

With this patch we get pretty far:

```
❯ holos generate platform holos
9:14AM INF platform.go:79 wrote platform.metadata.json version=0.84.1 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=/Users/jeff/Holos/holos-infra/saas2/platform.metadata.json
9:14AM INF platform.go:91 generated platform holos version=0.84.1 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=/Users/jeff/Holos/holos-infra/saas2

❯ time holos render platform --concurrency 1 ./platform
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/eso-creds-manager cluster=management num=1 total=73 duration=212.546542ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/cert-letsencrypt cluster=management num=2 total=73 duration=110.363875ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/certificates cluster=management num=3 total=73 duration=154.642541ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/login/zitadel-certs cluster=management num=4 total=73 duration=115.132041ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/ecr-creds-manager cluster=management num=5 total=73 duration=162.559542ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/eks-pod-identity-webhook cluster=management num=6 total=73 duration=135.03ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/crossplane/crds cluster=management num=7 total=73 duration=296.536833ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/crossplane/controller cluster=management num=8 total=73 duration=146.730667ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/backstage/management/certs cluster=management num=9 total=73 duration=117.42625ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/external-secrets cluster=aws1 num=10 total=73 duration=170.574458ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/eso-creds-refresher cluster=aws1 num=11 total=73 duration=161.188625ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/secretstores cluster=aws1 num=12 total=73 duration=153.708458ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/ecr-creds-refresher cluster=aws1 num=13 total=73 duration=130.369166ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/gateway-api cluster=aws1 num=14 total=73 duration=2.078997458s
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/istio/base cluster=aws1 num=15 total=73 duration=145.869084ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/istio/mesh/cni cluster=aws1 num=16 total=73 duration=142.113125ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/istio/mesh/istiod cluster=aws1 num=17 total=73 duration=155.186375ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/istio/mesh/gateway cluster=aws1 num=18 total=73 duration=137.8775ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/istio/mesh/httpbin/backend cluster=aws1 num=19 total=73 duration=116.537458ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/istio/mesh/httpbin/routes cluster=aws1 num=20 total=73 duration=122.709875ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/pgo/crds cluster=aws1 num=21 total=73 duration=271.561666ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/pgo/controller cluster=aws1 num=22 total=73 duration=143.880292ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/login/zitadel-secrets cluster=aws1 num=23 total=73 duration=116.962167ms
9:14AM INF platform.go:52 ok render component version=0.84.1 path=components/login/zitadel-database cluster=aws1 num=24 total=73 duration=121.315875ms
9:14AM ERR could not execute version=0.84.1 code=unknown err="could not build /Users/jeff/Holos/holos-infra/saas2/components/login/zitadel-server: spec.components.helmChartList.0.resourcesFile: field not allowed" loc=builder.go:166
spec.components.helmChartList.0.resourcesFile: field not allowed:
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:106:9
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:106:27
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:118:3
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:122:4
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:125:4
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:125:43
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:162:10
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:165:37
    /Users/jeff/Holos/holos-infra/saas2/buildplan.cue:206:13
    /Users/jeff/Holos/holos-infra/saas2/components/login/zitadel-server/zitadel.cue:9:1
    /Users/jeff/Holos/holos-infra/saas2/components/login/zitadel-server/zitadel.cue:18:9
    /Users/jeff/Holos/holos-infra/saas2/components/login/zitadel-server/zitadel.cue:19:9
    /Users/jeff/Holos/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:31:8
    /Users/jeff/Holos/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:36:15
    /Users/jeff/Holos/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:42:19
    /Users/jeff/Holos/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:42:22
    /Users/jeff/Holos/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:48:18
    /Users/jeff/Holos/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/helm_go_gen.cue:12:13
    /Users/jeff/Holos/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/helm_go_gen.cue:13:2
9:14AM ERR could not execute version=0.84.1 code=unknown err="could not render component: exit status 1" loc=platform.go:48
holos render platform --concurrency 1 ./platform  6.62s user 1.22s system 133% cpu 5.878 total
```
2024-06-30 09:14:52 -07:00
Jeff McCune
accf80200f (#189) Fix pod-identity-webhook Helm chart for v1alpha2
The pod identity webhook component fails to render with v1alpha2.  This
patch fixes the problem by providing concrete values for enableHooks and
the namespace of the helm chart holos component.

The namespace is mainly necessary to render the ArgoCD Application
resource along side the helm chart output.
2024-06-30 08:18:58 -07:00
Jeff McCune
4522ee1d4e (#189) Working eso-creds-manager with v1alpha2
With this patch the eso-creds-manager component renders correctly.  This
is a `#Kubernetes` type build plan which uses the
spec.components.resources map to manage resources.

The only issue was needing to provide the namespace to the nested holos
component inside the BuildPlan.

The ArgoCD Application resource moves to the DeployFiles field of a
separate holos component in the same build plan at
spec.components.resources.argocd.  For this reason a separate Result
object is no longer necessary inside of the Holos cli for the purpose of
managing Flux or ArgoCD gitops.  The CUE code can simply inline whatever
gitops resources it wants and the holos cli will write the files
relative to the cluster specific deploy directory.

Result:

```
❯ holos render component --cluster-name management components/eso-creds-manager
2:55PM INF result.go:195 wrote deploy file version=0.84.1 path=deploy/clusters/management/gitops/eso-creds-manager.application.gen.yaml bytes=350
2:55PM INF render.go:92 rendered eso-creds-manager version=0.84.1 cluster=management name=eso-creds-manager status=ok action=rendered
```
2024-06-29 14:55:53 -07:00
Jeff McCune
313ebc6817 (#189) README 2024-06-29 08:04:51 -07:00
Jeff McCune
e0f439515f (#189) Fix holos render platform for v1alpha2
Previously holos render platform failed for the holos platform.  The issue was
caused by the deployFiles field moving from the BuildPlan down to
HolosComponent.

This patch fixes the problem by placing the ArgoCD Application resource into a
separate Resources entry of the BuildPlan.  The sole purpose of this additional
entry in the Resources map is to produce the Application resource along side
any other components which are part of the build plan.
2024-06-29 07:32:57 -07:00
Jeff McCune
caa7560ab9 (#189) Fix Helm.Chart.namespace: field not allowed
Fixes:

```
4:19PM ERR could not execute version=0.84.1 code=unknown err="could not build /home/jeff/workspace/holos-run/holos-infra/saas2/platform: #Helm.Chart.namespace: field not allowed" loc=platform.go:52
    /home/jeff/workspace/holos-run/holos-infra/saas2/buildplan.cue:106:9
    /home/jeff/workspace/holos-run/holos-infra/saas2/buildplan.cue:108:3
    /home/jeff/workspace/holos-run/holos-infra/saas2/buildplan.cue:118:3
    /home/jeff/workspace/holos-run/holos-infra/saas2/buildplan.cue:118:43
    /home/jeff/workspace/holos-run/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/buildplan_go_gen.cue:48:18
    /home/jeff/workspace/holos-run/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/helm_go_gen.cue:12:13
    /home/jeff/workspace/holos-run/holos-infra/saas2/cue.mod/gen/github.com/holos-run/holos/api/core/v1alpha2/helm_go_gen.cue:13:2
```
2024-06-28 16:21:13 -07:00
Jeff McCune
bbcf280da7 (#189) Refactor v1alpha2 API
Previously methods were defined on the API objects in the v1alpha1 API.
The API should be data structures only.  This patch refactors the
methods responsible for orchestrating the build plan to pull them into
the internal render package.

The result is the API is cleaner and has no methods.  The render package
has corresponding data structures which simply wrap around the API
structure and implement the methods to render and return the result to
the CLI.

This commit compiles, but it has not been tested at all.  It's almost
surely broken completely.
2024-06-28 16:16:12 -07:00
Jeff McCune
6d2daacb7b (#189) Split api into meta and core groups
Previously in v1alpha1, all Holos structs are located in the same
package.  This makes it difficult to focus on only the structs necessary
to transfer configuration data from CUE to the `holos` cli.

This patch splits the structs into `meta` and `core` where the core
package holds the structs end users should refer to and focus on.  Only
the Platform resource is in core now, but other BuildPlan types will be
added shortly.
2024-06-28 13:02:44 -07:00
Jeff McCune
62f96a2d6c (#189) Add Go Documentation Server
Run it with:

    godoc -http=:6060
2024-06-28 12:42:34 -07:00
Jeff McCune
50f414d520 (#189) Platform v1alpha2
This patch moves the top level Platform API resource to v1alpha2 so it's
well documented using go docs.
2024-06-28 12:33:45 -07:00
Jeff McCune
882f3894f3 (#189) Clean up unused packages 2024-06-28 10:04:38 -07:00
Jeff McCune
30ddde7b49 (maint) Add make image to make help
Previously it wasn't clear how to build the image, wasn't showing up in
make help.
2024-06-24 20:48:47 -07:00
Jeff McCune
5cced6fb51 Version 0.84.0 2024-06-24 20:40:00 -07:00
Jeff McCune
a82ebf43b6 Merge pull request #187 from holos-run/jeff/180-backstage-component
(#180) Configure GitHub Apps Discovery
2024-06-24 20:38:17 -07:00
Jeff McCune
ebb6d6205a (#180) Configure GitHub Apps Discovery
Previously Backstage was not configured to integrate with GitHub.  The
integration is necessary for Backstage to automatically discover
resources in a GitHub organization and import them into the Catalog.

This patch adds a new platform model form field and section for the
primary GitHub organization name of the platform.  Additional GitHub
organizations can be added in the future, Backstage supports them.

The result is Backstage automatically scans public and private
repositories and adds the information in `catalog-info.yaml` to the UI.
2024-06-24 20:35:20 -07:00
Jeff McCune
58950c469a (#180) Manage default-istio ServiceAccount
Previosly the gateway ArogCD Application resource is out of sync because
the `default-istio` `ServiceAccount` is not in the git repository
source.  Argo would prune the service account on sync which is a problem.

This patch manages the service account so the Application can be synced
properly.
2024-06-13 06:04:10 -07:00
Jeff McCune
0eebdaf0c7 (#180) Fix authpolicy component after generate
Previously the holos render platform command fails with the following
error when giving a demo after the generate platform step.

This patch updates the internal generated holos platform to the latest
version.

Running through the demo is successful now.

```
holos logout
holos login
holos register user
holos generate platform holos
holos pull platform config .
holos render platform ./platform
```
2024-06-13 05:51:47 -07:00
Jeff McCune
54e2f28f4c (#179) Double check if the error group is done.
I'm not sure if we should check in the loop, in the go routine, or in
both places.  Double check in both cases just to be sure we're not doing
extra unnecessary work.
2024-06-06 15:51:16 -07:00
Jeff McCune
d4d50ef12b (#179) Use errorgroup SetLimit to limit concurrency
Previously a channel was used to limit concurrency.  This is more
difficult to read and comprehend than the inbuilt errorgroup.SetLimit
functionality.

This patch uses `errgroup.`[Group.SetLimit()][1] to limit concurrency,
avoid leaking go routines, and avoid unnecessary work.

[1]: https://pkg.go.dev/golang.org/x/sync/errgroup#Group.SetLimit
2024-06-06 15:23:49 -07:00
Jeff McCune
075f2b16a4 Merge pull request #179 from holos-run:nate/concurrency
Add concurrency to 'holos render platform'
2024-06-06 15:10:50 -07:00
Nate McCurdy
6f8008a53c Add concurrency to 'holos render platform'
This adds concurrency to the 'holos render platform' command so platform
components are rendered in less time than before.

Default concurrency is set to `min(runtime.NumCPU(), 8)`, which is the
lesser of 8 or the number of CPU cores. In testing, I found that past 8,
there are diminishing or negative returns due to memory usage or
rendering each component.

In practice, this reduced rendering of the saas platform components from
~90s to ~28s on my 12-core macbook pro.

This also changes the key name of the Helm Chart's version in log lines
from `version` to `chart_version` since `version` already exists and
shows the Holos CLI version.
2024-06-06 15:04:55 -07:00
Jeff McCune
0618b52bae (#181) Add AuthorizationPolicy resources for admin interfaces
Previously, when a user registered and logged into the holos app server,
they were able to reach admin interfaces like
https://argocd.admin.example.com

This patch adds AuthorizationPolicy resources governing the whole
cluster.  Users with the prod-cluster-{admin,edit,view} roles may access
admin services like argocd.

Users without these roles are blocked with RBAC: access denied.

In ZITADEL, the Holos Platform project is granted to the CIAM
organization without granting the prod-cluster-* roles, so there's no
possible way a CIAM user account can have these roles.
2024-06-06 14:57:48 -07:00
Jeff McCune
f1951c5db3 (#178) Add holos push platform model command
Previously there wasn't a good way to populate the platform model in the
database after building a new instance of holos server.

With this patch, the process to reset clean is:

```
export HOLOS_SERVER=https://dev.app.holos.run:443
grpcurl -H "x-oidc-id-token: $(holos token)" ${HOLOS_SERVER##*/} holos.user.v1alpha1.SystemService.DropTables
grpcurl -H "x-oidc-id-token: $(holos token)" ${HOLOS_SERVER##*/} holos.system.v1alpha1.SystemService.SeedDatabase
```

Then populate the form and model:

```
holos push platform form .
holos push platform model .
```

The `platform.config.json` file stored in version control is pushed to
the holos server and stored in the database.  This makes it nice and
easy to reset entirely, or move to another service url.
2024-06-05 15:38:55 -07:00
Jeff McCune
dad12acd8d (#178) Seed the Holos Platform itself
Previously this would have needed to be created in pgAdmin.
2024-06-05 14:17:31 -07:00
Jeff McCune
a4503e076f (#178) Add make image task to push the container image
Previously there wasn't an easy way to make the container image and
publish it.  This adds a simple `make image` task to build and push the
image.
2024-06-05 14:03:31 -07:00
Jeff McCune
09ddd339b8 (#178) Update user ids for SeedDatabase rpc
Need them to match the new login issuer.
2024-06-05 13:57:52 -07:00
Jeff McCune
bc94f4b6b8 (#178) Login to https://login.holos.run
Previously the default oidc issuer was to one of the kubernetes clusters
running in my basement.  This patch changes the issuer to the production
ready issuer running in EKS.
2024-06-05 13:42:37 -07:00
Jeff McCune
564406f60f (#178) Add app.example.com HTTPRoute for holos server
Previously the holos server Service was not exposed.

This patch exposes the holos service with an HTTPRoute behind the auth
proxy.  Holos successfully authenticates the user with the
x-oidc-id-token header set by the default Gateway.

---

Add dev-holos-infra and dev-holos-app

Previously the PostgresCluster and the holos server Deployment are not
managed on the aws2 cluster.

This patch is a start, but the Deployment does not yet start.  We need
to pass an option for the oidc issuer.

---

Add namespaces and cert for prod-holos, dev-holos, jeff-holos

Previously we didn't have a place to deploy holos server.  This patch
adds a namespace, creates a Gateway listener, and binds the tls certs
for app.example.com and *.app.example.com to the listeners.

In addition, cluster specific endpoints of *.app.aws2.example.com,
*.app.aws1.example.com, etc. are created to provide dev environment
urls. For example jeff.app.aws2.example.com is my personal dev hostname.
2024-06-05 13:15:11 -07:00
Jeff McCune
7845ce62e0 (#178) Update buf with make tools
Previously go releaser was failing because buf has been updated again.
2024-06-03 11:10:42 -07:00
Jeff McCune
a1542752b7 (#178) Add ArgoCD Application resources for each build plan
Previously holos render platform ./platform did not render any GitOps
resources for Flux or ArgoCD.

This patch uses the new DeployFiles field in holos v0.83.0 to write an
Application resource for every component BuildPlan listed in the
platform.
2024-06-03 10:33:05 -07:00
Jeff McCune
7956475363 (#178) Add BuildPlan deployFiles field
Previously, each BuildPlan has no clear way to produce an ArgoCD
Application resource.  This patch provides a general solution where each
BuildPlan can provide arbitrary files as a map[string]string where the
key is the file path relative to the gitops repository `deploy/` folder.
2024-06-03 10:00:35 -07:00
Jeff McCune
004ed56591 (#178) Add ArgoCD repository credentials
Previously ArgoCD has no ssh credentials to connect to GitHub.  This
patch adds an ssh ed25519 key as a secret in the management cluster.
The secret is synced to the workload clusters using an ExternalSecret
with the proper label for ArgoCD to find and load it for use with any
application that references the Git URL.
2024-06-02 15:58:35 -07:00
Jeff McCune
d497df3c27 (#178) Add ArgoCD RBAC Policy
Previously a logged in user could not modify anything in ArgoCD.  With
this patch users who have been granted the prod-cluster-admin role in
ZITADEL are granted the admin role in ArgoCD.
2024-06-02 15:07:27 -07:00
Jeff McCune
3a8d46234f (#178) Add ArgoCD
Previously ArgoCD was present in the platform configuration, but not
functional.  This patch brings ArgoCD fully up, integrated with the
service mesh, auth proxy, and SSO at
https://argocd.admin.clustername.example.com/

The upstream [helm chart][1] is used instead of the kustomize install
method.  We had existing prior art integrating the v6 helm chart with
the holos platform identity provider, so we continue with the helm
chart.

CRDs are still managed with the kustomize version.  The CRDs need to be
kept in sync.  It's possible to generate the kustomization.yaml file
from the same version value as is used by the helm chart, but we don't
for the time being.

[1]: https://github.com/argoproj/argo-helm/tree/argo-cd-7.1.1/charts/argo-cd
2024-06-02 14:35:36 -07:00
Jeff McCune
4d24dc5149 (#178) Add authpolicy component for RequestAuthentication
Previously, no RequestAuthentication or AuthorizationPolicy resources
govern the default Gateway.  This patch adds the resources and
configures the service mesh with the authproxy as an ExtAuthZ provider
for CUSTOM AuthorizationPolicy rules.

This patch also fixes a bug in the zitadel-server component where
resources from the upstream helm chart did not specify a namespace.
Kustomize is used as a post processor to force all resources into the
zitadel namespace.

Add multiple HTTPRoutes to validate http2 connection reuse

This patch adds multiple HTTPRoute resources which match
*.admin.example.com  The purpose is to validate http2 connections are
reused properly with Chrome.

With this patch no 404 no route errors are encountered when navigating
between the various httpbin{1,2,3,4} urls.

Add note backupRestore will trigger a restore

The process of configuring ZITADEL to provision from a datasource will
cause an in-place restore from S3.  This isn't a major issue, but users
should be aware data added since the most recent backup will be lost.
2024-06-02 09:41:57 -07:00
Jeff McCune
8eb7fbf7dc (#178) Move httpbin HTTPRoute resources to namespace istio-gateways
Previously, HTTPRoute resources were in the same namespace as the
backend service, httpbin in this case.  This doesn't follow the default
behavior of a Gateway listener only allowing attachment from HTTPRoute
resources in the same namespace as the Gateway.

This also complicates intercepting the authproxy path prefix and sending
it to the authproxy.  We'd need to add a ReferenceGrant in the authproxy
namespace, which seems backwards and dangerous because it would grant
the application developer the ability to route requests to all Services
in the istio-gateways namespace.

This patch enables Cluster Operators to manage the HTTPRoute resources
and direct the auth proxy path prefix of `/holos/authproxy` to the auth
proxy Service in the same namespace.

ReferenceGrant resources are used to enable the HTTPRoute backend
references.

When an application developer needs to manage their own HTTPRoute, as is
the case for ZITADEL, a label selector may be used and will override
less specific HTTPRoute hostsnames in the istio-gateways namespace.
2024-06-01 21:18:47 -07:00
Jeff McCune
ffeeb7c553 (#178) Add authproxy Deployment
With redis.  The auth proxy authenticates correctly against zitadel
running in the same cluster.  Validated by visiting
https://httpbin.admin.clustername.example.com/holos/authproxy

Visiting
https://httpbin.admin.clustername.example.com/holos/authproxy/auth
returns the id token in the response header, visible in the Chrome
network inspector.  The ID token works as expected from multiple orgs
with project grants in ZITADEL from the Holos org to the OIS org.

This patch doesn't fully implement the auth proxy feature.
AuthorizationPolicy and RequestAuthentication resources need to be
added.

Before we do so, we need to move the HTTPRoute resources into the
gateway namespace so all of the security policies are in one place and
to simplify the process of routing requests to two backends, the
authproxy and the backend server.
2024-06-01 20:12:35 -07:00
Jeff McCune
c3c174155c (#178) Add httpbin{1,2,3,4} HTTPRoutes to validate http2 connection reuse
This patch adds multiple HTTPRoute resources which match
*.admin.example.com  The purpose is to validate http2 connections are
reused properly with Chrome.

With this patch no 404 no route errors are encountered when navigating
between the various httpbin{1,2,3,4} urls.
2024-06-01 12:44:33 -07:00
Jeff McCune
2c2d2a9fd9 (#178) Add Namespaces documentation
Describe how to manage a new namespace to build a component in.
2024-06-01 09:43:32 -07:00
Jeff McCune
d692e2a6d5 (#178) Split subdomain certs into two certs
Problem:
Istio 1.22 with Gateway API and HTTPRoute is mis-routing HTTP2 requests
when the tls certificate has two dns names, for example
login.example.com and *.login.example.com.

When the user visits login.example.com and then tries to visit
other.login.example.com with Chrome, the connection is re-used and istio
returns a 404 route not found error even though there is a valid and
accepted HTTPRoute for *.login.example.com

This patch attempts to fix the problem by ensuring certificate dns names
map exactly to Gateway listeners.  When a wildcard cert is used, the
corresponding Gateway listener host field exactly matches the wildcard
cert dns name so Istio and envoy should not get confused.
2024-06-01 09:30:47 -07:00
Jeff McCune
e4cebddd0c (#178) Make aws2 the primary cluster 2024-05-31 14:01:11 -07:00
Jeff McCune
0e48537d65 (#178) Add zitadel-server component
This patch adds the ZITADEL server component, which deploys zitadel from
a helm chart.  Kustomize is used heavily to patch the output of helm to
make the configuration fit nicely with the holos platform.

With this patch the two Jobs that initialize the database and setup
ZITADEL run successfully.  The ZITADEL deployment starts successfully.

ZITADEL is accessible at https://login.example.com/ with the default
admin username of `zitadel-admin@zitadel.login.example.com` and password
`Password1!`.

Use grant.holos.run/subdomain.admin: "true" for HTTPRoute

This patch clarifies the label that grants httproute attachment for a
subdomain Gateway listener to a namespace.

Fix istio-base holos component name

Was named `base` which is the chart name, not the holos component name.
2024-05-31 13:47:03 -07:00
Jeff McCune
a461a96b9c (#178) Add ZITADEL crunchy pgo PostgresCluster
This patch adds the postgres clusters and a few console form controls to
configure how backups are taken and if the postgres cluster is
initialized from an existing backup or not.

The pgo-s3-creds file is manually created at this time.  It looks like:

    ❯ holos get secret -n zitadel pgo-s3-creds --print-key s3.conf
    [global]
    repo2-cipher-pass=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    repo2-s3-key=KKKKKKKKKKKKKKKKKKKK
    repo2-s3-key-secret=/SSSSSSS/SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
    repo3-cipher-pass=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    repo3-s3-key=KKKKKKKKKKKKKKKKKKKK
    repo3-s3-key-secret=/SSSSSSS/SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

The s3 key and secret are credentials to read / write to the bucket.
The cipher pass is a random string for client side encryption.  Generate
it with `tr -dc A-Za-z0-9 </dev/urandom | head -c 64`
2024-05-30 11:33:00 -07:00
Jeff McCune
9524c4f7c3 (#178) Add crunchy postgres operator
Needed for ZITADEL and Holos Server.  Intended for ephemeral dev
environments, but may also try it in staging while we wait for RDS.
2024-05-29 12:03:05 -07:00
Jeff McCune
64b04d9cfd (#178) Add Gateway listener for login.example.com
This patch is foundational work for the ZITADEL login service.

This patch adds a tls certificate with names *.login.example.com and
login.example.com, a pair of listeners attached to the certificate in
the `default` Gateway, and the ExternalSecret to sync the secret from
the management cluster.

The zitadel namespace is managed and has the label
holos.run/login.grant: "true" to grant HTTPRoute attachment from the
zitadel namespace to the default Gateway in the istio-gateways
namespace.
2024-05-29 09:27:08 -07:00
Jeff McCune
b419ad8caf (#178) Add HTTPRoute for httpbin.admin.aws1.example.com
With this change, https://httpbin.admin.aws1.example.com works as
expected.

PROXY protocol is configured on the AWS load balancer and the istio
gateway.  The istio gateway logs have the correct client source ip
address and x-forwarded-for headers.

Namespaces must have the holos.run/admin.grant: "true" label in order to
attach an HTTPRoute to the admin section of the default Gateway.

The TLS certificate is working as expected and hopefully does not suffer
from the NR route not found issued encountered with the Istio Gateway
API.
2024-05-28 21:10:18 -07:00
Jeff McCune
8036c17916 (#178) Add istiod and gateway components
This patch gets the istio-ingressgateway up and running in AWS with
minimal configuration.  No authentication or authorization policies have
been migrated from previous iterations of the platform.  These will be
handled in subsequent iterations.

Connectivity to a backend service like httpbin has not yet been tested.
This will happen in a follow up as well using /httpbin path prefixes on
existing services like argocd to conserve certificate resources.
2024-05-28 14:37:25 -07:00
Jeff McCune
220d498be0 (#178) Define a #IngressCertificate
This is the standard way to issue public facing certificates.  Be aware
of the 50 cert limit per week from LetsEncrypt.  We map names to certs
1:1 to avoid http2 connection reuse issues with istio.
2024-05-28 13:15:14 -07:00
Jeff McCune
0f5b6a2d6e (#178) Add istio 1.22.0 base component 2024-05-28 13:08:34 -07:00
Jeff McCune
36369d75c7 (#178) Add argocd.admin.aws1.holos.run cert
Manage certificates on a project basis similar to how namespaces
associated with each project are managed.

Manage the Certificate resources on the management cluster in the
istio-ingress namespace so the tls certs can be synced to the workload
clusters.
2024-05-28 11:50:31 -07:00
Jeff McCune
059b8283fd (#178) Add cert-letsencrypt component for holos management cluster
The secret needs to be manually provisioned for this to work since the
management cluster does not sync secrets from any other external
cluster.
2024-05-26 09:56:46 -07:00
Jeff McCune
386eb2452a (#178) Add cert-manager to the holos platform
This patch adds cert-manager on all clusters.  On the management cluster
cert manager is scheduled on spot instances to reduce cost.
2024-05-26 09:29:15 -07:00
Jeff McCune
38e9a97fd2 (#178) Add secretstores holos platform component
The secretstores component is critical and provides the mechanism to
securely fetch Secret resources from the Management Cluster.
The holos server and configuration code stored in version control
contains only ExternalSecret references, no actual secrets.

This component adds a `default` `SecretStore` to each management
namespace which uses the `eso-reader` service account token to
authenticate to the management cluster.  This service account is limited
to reading secrets within the namespace it resides in.

For example:

```yaml
---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: default
  namespace: external-secrets
spec:
  provider:
    kubernetes:
      auth:
        token:
          bearerToken:
            key: token
            name: eso-reader
      remoteNamespace: external-secrets
      server:
        caBundle: Long Base64 encoded string
        url: https://34.121.54.174
```
2024-05-25 15:02:06 -07:00
Jeff McCune
ecca40e9d5 (#178) Add holos platform eso-creds-manager
This patch adds the `eso-creds-manager` component which needs to be
applied to the management cluster prior to the `eso-creds-refreher`
component being applied to workload clusters.

The manager component configures rbac to allow the creds-refresher job
to complete.

This patch also adjusts the behavior to only create secrets for the
eso-reader account by default.

Namespaces with the label `holos.run/eso.writer=true` will also have an
eso-writer secret provisioned in their namespace, allowing secrets to be
written back to the management cluster.  This is intended for the
PushSecret resource.
2024-05-24 15:09:59 -07:00
Jeff McCune
9d08e27e31 (#178) Add cue.mod/gen/k8s.io/api/batch/v1 2024-05-23 16:33:58 -07:00
Jeff McCune
969bf5e867 (#178) Import k8s rbac api
cue get go k8s.io/api/rbac/v1beta1
cue get go k8s.io/api/rbac/v1
2024-05-23 16:26:24 -07:00
Jeff McCune
3b5f28f4df (#178) Fix holos generate writing executable files
Adhere to the umask to allow group writable or world writable, but do
not set the execute bit.
2024-05-23 11:37:04 -07:00
Jeff McCune
df5619f988 (#178) Add ArgoCD schematic and component to holos
Add the ArgoCD component which is a good example of how to wrap a plain
kustomize kustomization.yaml file with Holos.
2024-05-23 11:18:29 -07:00
Jeff McCune
a6d8383176 (#178) Do not write flux kustomization if empty
If the holos component returns no data for the flux kustomization, don't
bother writing an useless empty file.
2024-05-23 10:56:08 -07:00
Jeff McCune
dbc7e374cd (#178) Update buf 2024-05-23 09:37:46 -07:00
Jeff McCune
d81729857b (#178) v0.81.2 for holos
Use v0.81.2 to build out the holos platform.  Once we have the
components structured fairly well we can circle back around and copy the
components to schematics.  There's a bit of friction regenerating the
platform from schematic each time.
2024-05-23 09:14:27 -07:00
Jeff McCune
d3d8a7b73c (#178) Shape _Namespaces to corev1.#Namespace
Eliminate the need for a for loop by having _Namespaces be a struct of
name to k8s.io/api/core/v1.#Namespace
2024-05-23 09:12:08 -07:00
Jeff McCune
d9e6776b95 (#178) npm upgrade 2024-05-23 06:41:10 -07:00
Jeff McCune
bde98faffa (#178) Use private fields to store data
Using CUE definitions like #Platform to hold data is confusing.  Clarify
the use of fields, definitions like #Platform define the shape (schema)
of the data while private fields like _Platform represent and hold the
data.
2024-05-23 06:38:52 -07:00
Jeff McCune
c2847554e0 (#178) Add namespaces to holos platform 2024-05-22 17:04:47 -07:00
Jeff McCune
9411a65dd8 (#178) Add namespaces component schematic
The first thing most platforms need to do is come up with a strategy for
managing namespaces across multiple clusters.

This patch defines #Namespaces in the holos platform and adds a
namespaces component which loops over all values in the #Namespaces
struct and manages a kubernetes Namespace object.

The platform resource itself loops over all clusters in the platform to
manage all namespaces across all clusters.

From a blank slate:

```
❯ holos generate platform holos
4:26PM INF platform.go:79 wrote platform.metadata.json version=0.82.0 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=/home/jeff/workspace/holos-run/holos-infra/saas/platform.metadata.json
4:26PM INF platform.go:91 generated platform holos version=0.82.0 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=/home/jeff/workspace/holos-run/holos-infra/saas

❯ holos pull platform config .
4:26PM INF pull.go:64 pulled platform model version=0.82.0 server=https://jeff.app.dev.k2.holos.run:443 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc
4:26PM INF pull.go:75 saved platform config version=0.82.0 server=https://jeff.app.dev.k2.holos.run:443 platform_id=018fa1cf-a609-7463-aa6e-fa53bfded1dc path=platform.config.json

❯ (cd components && holos generate component cue namespaces)
4:26PM INF component.go:147 generated component version=0.82.0 name=namespaces path=/home/jeff/workspace/holos-run/holos-infra/saas/components/namespaces

❯ holos render platform ./platform/
4:26PM INF platform.go:29 ok render component version=0.82.0 path=components/namespaces cluster=management num=1 total=2 duration=464.055541ms
4:26PM INF platform.go:29 ok render component version=0.82.0 path=components/namespaces cluster=aws1 num=2 total=2 duration=467.978499ms
```

The result:

```sh
cat deploy/clusters/management/components/namespaces/namespaces.gen.yaml
```

```yaml
---
metadata:
  name: holos
  labels:
    kubernetes.io/metadata.name: holos
kind: Namespace
apiVersion: v1
```
2024-05-22 16:32:59 -07:00
Jeff McCune
9c1165e77e (#178) Save platform.config.json with multiple lines 2024-05-22 14:10:28 -07:00
Jeff McCune
a02c7a4015 (#178) Fix the PlatformService CreatePlatform rpc
Without this patch the
holos.platform.v1alpha1.PlatformService.CreatePlatform doesn't work as
expected.  The Platform message is used which incorrectly requires a
client supplied id which is ignored by the server.

This patch allows the creation of a new platform by reusing the update
operation as a mutation that applies to both create and update.  Only
modifiable fields are part of the PlatformMutation message.
2024-05-22 12:39:24 -07:00
Jeff McCune
bdcde88e6f (#175) Add git describe to --version output
Much easier to track changes between releases.
2024-05-21 13:21:27 -07:00
Jeff McCune
a32b100192 (#175) Output at the end
Flip the let definitions to before their use to avoid confusing /
distracting users who are just getting started.

User feedback from Nate.
2024-05-21 13:03:22 -07:00
Jeff McCune
670d716403 (#175) Add podinfo oci example
This patch adds to more example helm chart based components.  podinfo
installs as a normal https repository based helm chart.  podinfo-oci
uses an oci image to manage the helm chart.

The way holos handls OCI images is subtle, so it's good to include an
example right out of the chute.  Github actions uses OCI images for
example.
2024-05-21 12:36:45 -07:00
Jeff McCune
bba3895f35 (#175) Add holos generate component helm command
This patch adds a schematic to generate a holos component that wraps a
helm chart.  The cert-manager chart is the current example.

Usage:

```bash
set -euo pipefail

rm -rf ~/holos/dev/bare
mkdir ~/holos/dev/bare
cd ~/holos/dev/bare

holos generate platform bare
holos pull platform config .
holos render platform ./platform/
(cd components && holos generate component helm cert-manager)
```

The chart builds:

```bash
holos build ./components/cert-manager | yq .
```

And renders:

```bash
holos render component ./components/cert-manager --cluster-name k2
find deploy -type f
```

```txt
9:41PM INF render.go:83 rendered cert-manager version=0.81.1 cluster=k2 status=ok action=rendered name=cert-manager
deploy/clusters/k2/holos/components/cert-manager-kustomization.gen.yaml
deploy/clusters/k2/components/cert-manager/cert-manager.gen.yaml
```
2024-05-21 11:05:53 -07:00
Jeff McCune
9e60ddbe85 (#175) Add holos generate component cue command
This patch adds a command to generate CUE based holos components from
examples embedded in the executable.  The examples are passed through
the go template rendering engine with values pulled from flags.

Each directory in the embedded filesystem becomes a unique command for
nice tab completion.  The `--name` flag defaults to "example" and is the
resulting component name.

A follow up patch with more flags will set the stage for a Helm
component schematic.

```
holos generate component cue minimal
```

```txt
3:07PM INF component.go:91 generated component version=0.80.2 name=example path=/home/jeff/holos/dev/bare/components/example
```
2024-05-20 15:10:54 -07:00
Jeff McCune
44334fca52 (#175) Fix lint 2024-05-20 12:39:43 -07:00
Jeff McCune
2e2ed398c6 (#175) Fix tests 2024-05-20 11:32:29 -07:00
Jeff McCune
34f2a52cb7 (#175) Add holos render platform command
Split holos render into component and platform.

This patch splits the previous `holos render` command into subcommands.
`holos render component ./path/to/component/` behaves as the previous
`holos render` command and renders an individual component.

The new `holos render platform ./path/to/platform/` subcommand makes
space to render the entire platform using the platform model pulled from
the PlatformService.

Starting with an empty directory:

```sh
holos register user
holos generate platform bare
holos pull platform config .
holos render platform ./platform/
```

```txt
10:01AM INF platform.go:29 ok render component version=0.80.2 path=components/configmap cluster=k1 num=1 total=1 duration=448.133038ms
```

The bare platform has a single component which refers to the platform
model pulled from the PlatformService:

```sh
cat deploy/clusters/mycluster/components/platform-configmap/platform-configmap.gen.yaml
```

```yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: platform
  namespace: default
data:
  platform: |
    spec:
      model:
        cloud:
          providers:
            - cloudflare
        cloudflare:
          email: platform@openinfrastructure.co
        org:
          displayName: Open Infrastructure Services
          name: ois
```
2024-05-20 10:41:24 -07:00
Jeff McCune
d3888a884f (#175) go mod tidy 2024-05-20 06:32:53 -07:00
Jeff McCune
3845871368 (#175) holos pull platform config
This patch adds a subcommand to pull the data necessary to construct a
PlatformConfig DTO.  The PlatformConfig message contains all of the
fields and values necessary to build a platform and the platform
components.  This is an alternative to holos passing multiple tags to
CUE.  The PlatformConfig is marshalled and passed once.

The platform config is also stored in the local filesystem in the root
directory of the platform.  This enables repeated local building and
rendering without making an rpc call.

The build / render pipeline is expected to cache the PlatformConfig once
at the start of the pipeline using the pull subcommand.
2024-05-19 08:27:21 -07:00
Jeff McCune
a3b2d19adb (#175) Render the platform with the model
The `holos render platform` command is unimplemented.  This patch
partially implements platform rendering by fetching the platform model
from the PlatformService and providing it to CUE using a tag.

CUE returns a `kind: Platform` resource to `holos` which will eventually
process a Buildlan for each platform component listed in the Platform
spec.

For now, however, it's sufficient to have the current platform model
available to CUE.
2024-05-18 11:40:30 -07:00
Jeff McCune
e4e7cd8c47 (#175) Make holos render --cluster-name flag optional
Problem:
Rendering the whole platform doesn't need a cluster name.

Solution:
Make the flag optional, do not set the cue tag if it's empty.

Result:
Holos renders the platform resource and proceeds to the point where we
need to implement the iteration over platform components, passing the
platform model to each one and rendering the component.
2024-05-17 15:48:36 -07:00
Jeff McCune
fb22e5521b (#175) Define the Platform resource in CUE
We need to output a kind: Platform resource from cue so holos can
iterate over each build plan.  The platform resource itself should also
contain a copy of the platform model obtained from the PlatformService
so holos can easily pass the model to each BuildPlan it needs to execute
to render the full platform.

This patch lays the groundwork for the Platform resource.  A future
patch will have the holos cli obtain the platform model and inject it as
a JSON encoded string to CUE.  CUE will return the Platform resource
which is a list of references to build plans.  Holos will then iterate
over each build plan, pass the model back in, and execute the build
plan.

To illustrate where we're headed, the `cue export` step will move into
`holos` with a future patch.

```
❯ holos register user
3:34PM INF register.go:77 user version=0.80.0 email=jeff@ois.run server=https://app.dev.k2.holos.run:443 user_id=018f8839-3d74-7e39-afe9-181ad2fc8abe org_id=018f8839-3d74-7e3a-918c-b36494da0115
❯ holos generate platform bare
3:34PM INF generate.go:79 wrote platform.metadata.json version=0.80.0 platform_id=018f8839-3d74-7e3b-8cb8-77a2c124d173 path=/home/jeff/holos/dev/bare/platform.metadata.json
3:34PM INF generate.go:91 generated platform bare version=0.80.0 platform_id=018f8839-3d74-7e3b-8cb8-77a2c124d173 path=/home/jeff/holos/dev/bare
❯ holos push platform form .
3:34PM INF push.go:70 pushed: https://app.dev.k2.holos.run:443/ui/platform/018f8839-3d74-7e3b-8cb8-77a2c124d173 version=0.80.0
❯ cue export ./platform/
{
    "metadata": {
        "name": "bare",
        "labels": {},
        "annotations": {}
    },
    "spec": {
        "model": {}
    },
    "kind": "Platform",
    "apiVersion": "holos.run/v1alpha1"
}
```
2024-05-17 15:34:56 -07:00
Jeff McCune
d2ae766ae3 Merge pull request #176 from holos-run/dependabot/go_modules/github.com/docker/docker-26.0.2incompatible
Bump github.com/docker/docker from 26.0.0+incompatible to 26.0.2+incompatible
2024-05-17 11:53:44 -07:00
dependabot[bot]
c0db949729 Bump github.com/docker/docker
Bumps [github.com/docker/docker](https://github.com/docker/docker) from 26.0.0+incompatible to 26.0.2+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](https://github.com/docker/docker/compare/v26.0.0...v26.0.2)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-05-17 18:52:51 +00:00
Jeff McCune
d2d4337ffd (#175) Improve url output
❯ holos push platform form .
11:49AM INF push.go:70 pushed: https://app.dev.k2.holos.run:443/ui/platform/018f87d1-7ca2-7e37-97ed-a06bcee9b442 version=0.79.0
2024-05-17 11:49:04 -07:00
Jeff McCune
b0ca04635e (#175) Update the client context when switching servers
When the holos server URL switches, we also need to update the client
context to get the correct org id.

Also improve quality of life by printing the url to the form when the
platform form is pushed to the server.

❯ holos push platform form .
11:41AM INF push.go:71 updated platform form version=0.79.0 server=https://app.dev.k2.holos.run:443 platform_id=018f87d1-7ca2-7e37-97ed-a06bcee9b442
11:41AM INF push.go:72 https://app.dev.k2.holos.run:443/ui/platform/018f87d1-7ca2-7e37-97ed-a06bcee9b442 version=0.79.0
2024-05-17 11:43:52 -07:00
Jeff McCune
198c66e6cd (#175) Fix tests
Not sure why this started failing, but it wasn't necessary.
2024-05-17 10:22:35 -07:00
Jeff McCune
24346b9a38 (#172) Deploy v0.79.0 to dev 2024-05-17 10:15:05 -07:00
Jeff McCune
0639562f1c (#175) go mod tidy 2024-05-17 10:09:40 -07:00
Jeff McCune
c1fa9cc531 (#175) Fix lint 2024-05-17 10:08:06 -07:00
Jeff McCune
18653534ad (#175) Add holos push platform form command
This sub-command renders the web app form from CUE code and updates the
form using the `holos.platform.v1alpha1.PlatformService/UpdatePlatform`
rpc method.

Example use case, starting fresh:

```
rm -rf ~/holos
mkdir ~/holos
cd ~/holos
```

Step 1: Login

```sh
holos login
```

```txt
9:53AM INF login.go:40 logged in as jeff@ois.run version=0.79.0 name="Jeff McCune" exp="2024-05-17 21:16:07 -0700 PDT" email=jeff@ois.run
```

Step 2: Register to create server side resources.

```sh
holos register user
```

```
9:52AM INF register.go:68 user version=0.79.0 email=jeff@ois.run user_id=018f826d-85a8-751d-81ee-64d0f2775b3f org_id=018f826d-85a8-751e-98dd-a6cddd9dd8f0
```

Step 3: Generate the bare platform in the local filesystem.

```sh
holos generate platform bare
```

```txt
9:52AM INF generate.go:79 wrote platform.metadata.json version=0.79.0 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 path=/home/jeff/holos/platform.metadata.json
9:52AM INF generate.go:91 generated platform bare version=0.79.0 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 path=/home/jeff/holos
```

Step 4: Push the platform form to the `holos server` web app.

```sh
holos push platform form .
```

```txt
9:52AM INF client.go:67 updated platform version=0.79.0 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 duration=73.62995ms
```

At this point the platform form is published and functions as expected
when visiting the platform web interface.
2024-05-17 09:51:36 -07:00
Jeff McCune
2b89c33067 (#175) Add holos orgid command
Makes it easier to work with grpcurl:

    grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"org_id":"'$(holos orgid)'"}' ${HOLOS_SERVER##*/} holos.platform.v1alpha1.PlatformService.ListPlatforms
2024-05-16 21:11:24 -07:00
Jeff McCune
aee26d9375 (#175) Set header User-Agent: holos/0.70.0 (go1.22.2)
Previously: User-Agent: connect-go/1.16.0 (go1.22.2)
2024-05-16 20:49:06 -07:00
Jeff McCune
7b04d492ab (#175) Set http.Server ReadHeaderTimeout
Upstream connectrpc recommends it.  Refer to
https://connectrpc.com/docs/faq#stream-error
2024-05-16 20:28:31 -07:00
Jeff McCune
8abd03e165 (#175) Log x-request-id and x-b3-trace headers
This patch logs the x-request-id header which makes it straight forward
to correlate the logs with the service mesh logs.

For example, select the request id from the gateway logs by copying the
log from the holos server logs.

```sh
kubectl -n istio-ingress logs -l app=istio-ingressgateway -f \
  | grep --line-buffered '^{' \
  | jq 'select(.request_id=="'d0867115-5795-4096-942e-5ac188cdf618'")'
```

```json
{
  "upstream_local_address": "10.244.1.51:44248",
  "x_forwarded_for": "192.168.2.21",
  "authority": "jeff.app.dev.k2.holos.run:443",
  "upstream_transport_failure_reason": null,
  "connection_termination_details": null,
  "response_code": 200,
  "duration": 6,
  "response_flags": "-",
  "upstream_service_time": "5",
  "upstream_cluster": "outbound|3000||holos.jeff-holos.svc.cluster.local",
  "upstream_host": "10.244.1.249:3000",
  "user_agent": "connect-go/1.16.0 (go1.22.2)",
  "requested_server_name": "jeff.app.dev.k2.holos.run",
  "request_id": "d0867115-5795-4096-942e-5ac188cdf618",
  "start_time": "2024-05-17T03:16:37.900Z",
  "method": "POST",
  "protocol": "HTTP/2",
  "downstream_local_address": "65.102.23.41:443",
  "path": "/holos.user.v1alpha1.UserService/GetUser",
  "bytes_sent": 159,
  "downstream_remote_address": "192.168.2.21:59564",
  "response_code_details": "via_upstream",
  "bytes_received": 0,
  "route_name": "holos-api"
}
```
2024-05-16 20:14:34 -07:00
Jeff McCune
2df843bc98 (#175) Link the generated platform to holos server
When the user generates a platform, we need to know the platform ID it's
linked to in the holos server.  If there is no platform with the same
name, the `holos generate platform` command should error out.

This is necessary because the first thing we want to show is pushing an
updated form to `holos server`.  To update the web ui the CLI needs to
know the platform ID to update.

This patch modifies the generate command to obtain a list of platforms
for the org and verify the generated name matches one of the platforms
  that already exists.

A future patch could have the `generate platform` command call the
`holos.platform.v1alpha1.PlatformService.CreatePlatform` method if the
platform isn't found.

Results:

```sh
holos generate platform bare
```

```txt
4:15PM INF generate.go:77 wrote platform.metadata.json version=0.77.1 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 path=/home/jeff/holos/platform.metadata.json
4:15PM INF generate.go:89 generated platform bare version=0.77.1 platform_id=018f826d-85a8-751f-96d0-0d2bf70df909 path=/home/jeff/holos
```

```sh
cat platform.metadata.json
```

```json
{
  "id": "018f826d-85a8-751f-96d0-0d2bf70df909",
  "name": "bare",
  "display_name": "Bare Platform"
}
```
2024-05-16 16:18:38 -07:00
Jeff McCune
be4d2c29a5 (#175) Log info message when generating a platform
holos generate platform bare
    2:11PM INF generate.go:55 generated platform bare version=0.77.1 path=/home/jeff/holos
2024-05-16 14:26:51 -07:00
Jeff McCune
8ce88bf491 (#175) Fix goreleaser
Buf was being automatically updated in the pipeline.
2024-05-16 14:00:37 -07:00
Jeff McCune
b05571a595 (#175) Go tidy and update package.json
For goreleaser
2024-05-16 13:41:47 -07:00
Jeff McCune
4edfc71d68 (#175) Log the grpc procedure at info level
This patch logs the service and rpc method of every request at Info
level.  The error code and message is also logged.  This gives a good
indication of what rpc methods are being called and by whom.
2024-05-16 11:43:20 -07:00
Jeff McCune
3049694a0a (#175) holos register user
This patch adds a `holos register user` command.  Given an authenticated
id token and no other record of the user in the database, the cli tool
use the API to:

 1. User is registered in `holos server`
 2. User is linked to one Holos Organization.
 3. Holos Organization has the `bare` platform.
 4. Holos Organization has the `reference` platform.
 5. Ensure `~/.holos/client-context.json` contains the user id and an
    org id.

The `holos.ClientContext` struct is intended as a light weight way to
save and load the current organization id to the file system for further
API calls.

The assumption is most users will have only one single org.  We can add
a more complicated config context system like kubectl uses if and when
we need it.
2024-05-16 10:51:40 -07:00
Jeff McCune
5860c5747b (#87) generate sub-command with embedded platform
This patch adds a generate subcommand that copies a platform embedded
into the executable to the local filesystem.  The purpose is to
accelerate initial setup with canned example platforms.

Two platforms are intended to start, one bare and one reference
platform.  The number of platforms embedded into holos should be kept
small (2-3) to limit our support burden.
2024-05-14 15:03:21 -07:00
Jeff McCune
d3c2d55706 (#172) Deploy v0.76.0 to dev 2024-05-14 13:28:19 -07:00
Jeff McCune
ac2ff47a9c (#172) Wire Version Info in the UI
This patch adds the GetVersion rpc method to
holos.system.v1alpha1.SystemService and wires the version information up
to the Web UI.

This is a good example to crib from later regarding fetching and
refreshing data from the web ui using grpc and field masks.
2024-05-14 11:50:06 -07:00
Jeff McCune
9a2773c618 (#171) Refactor API to use FieldMasks
This patch refactors the API following the [API Best Practices][api]
documentation.  The UpdatePlatform method is modeled after a mutating
operation described [by Netflix][nflx] instead of using a REST resource
representation.  This makes it much easier to iterate over the fields
that need to be updated as the PlatformUpdateOperation is a flat data
structure while a Platform resource may have nested fields.  Nested
fields are more complicated and less clear to handle with a FieldMask.

This patch also adds a snapckbar message on save.  Previously, the save
button didn't give any indication of success or failure.  This patch
fixes the problem by adding a snackbar message that pop up at the bottom
of the screen nicely.

When the snackbar message is dismissed or times out the save button is
re-enabled.

[api]: https://protobuf.dev/programming-guides/api/
[nflx]: https://netflixtechblog.com/practical-api-design-at-netflix-part-2-protobuf-fieldmask-for-mutation-operations-2e75e1d230e4

Examples:

FieldMask for ListPlatforms

```
grpcurl -H "x-oidc-id-token: $(holos token)" -d @ ${HOLOS_SERVER##*/} holos.platform.v1alpha1.PlatformService.ListPlatforms <<EOF
{
  "org_id": "018f36fb-e3f7-7f7f-a1c5-c85fb735d215",
  "field_mask": { "paths": ["id","name"] }
}
EOF
```

```json
{
 "platforms": [
   {
     "id": "018f36fb-e3ff-7f7f-a5d1-7ca2bf499e94",
     "name": "bare"
   },
   {
     "id": "018f6b06-9e57-7223-91a9-784e145d998c",
     "name": "gary"
   },
   {
     "id": "018f6b06-9e53-7223-8ae1-1ad53d46b158",
     "name": "jeff"
   },
   {
     "id": "018f6b06-9e5b-7223-8b8b-ea62618e8200",
     "name": "nate"
   }
 ]
}
```

Closes: #171
2024-05-13 16:20:20 -07:00
Jeff McCune
51b6575d9f (#171) Refactor to API Best Practices
This patch refactors the API to be resource-oriented around one service
per resource type.  PlatformService, OrganizationService, UserService,
etc...

Validation is improved to use CEL rules provided by [protovalidate][1].

Place holders for FieldMask and other best practices are added, but are
unimplemented as per [API Best Practices][2].

The intent is to set us up well for copying and pasting solid existing
examples as we add features.

With this patch the server and web app client are both updated to use
the refactored API, however the following are not working:

 1. Update the model.
 2. Field Masks.

[1]: https://buf.build/bufbuild/protovalidate
[2]: https://protobuf.dev/programming-guides/api/
2024-05-10 15:55:41 -07:00
Jeff McCune
68a43f0682 (#167) Add holos rpc platform-model command
This command is just a prototype of how to fetch the platform model so
we can make it available to CUE.

The idea is we take the data from the holos server and write it into a
CUE `_Platform` struct.  This will probably involve converting the data
to CUE format and nesting it under the platform struct spec field.
2024-05-08 16:34:00 -07:00
Jeff McCune
9da88c4d1b (#169) ZITADEL ServerError - PGBouncer DNS
Add runbook notes.  Root cause and permanent solution have not been
identified yet.
2024-05-08 12:04:50 -07:00
Jeff McCune
19df2ec0fb (#167) Bump dev deployment to 0.74.0 2024-05-07 16:58:03 -07:00
Jeff McCune
bac7aec0ba (#167) Restructure the bare platform
This patch restructures the bare platform in preparation for a
`Platform` kind of output from CUE in addition to the existing
`BuildPlan` kind.

This patch establishes a pattern where our own CUE defined code goes
into the two CUE module paths:

1. `internal/platforms/cue.mod/gen/github.com/holos-run/holos/api/v1alpha1`
2. `internal/platforms/cue.mod/pkg/github.com/holos-run/holos/api/v1alpha1`
3. `internal/platforms/cue.mod/usr/github.com/holos-run/holos/api/v1alpha1`

The first path is automatically generated from Go structs.  The second
path is where we override and provide additional cue level integration.

The third path is reserved for the end user to further refine and
constrain our definitions.
2024-05-07 16:53:00 -07:00
Jeff McCune
42f916af41 (#164) Use quay.io/holos/oauth2-proxy:v7.6.0-1-g77a03ae2
Custom build to set samesite=none on the csrf cookie.
2024-05-06 16:18:32 -07:00
Jeff McCune
47a5e237e0 (#162) Lint go, typescript, and proto3 files
This patch adds lint coverage for proto3 and typescript to keep our code
reasonably clean.  The go linter was already enabled.
2024-05-06 14:17:08 -07:00
Jeff McCune
1279e2351a (#162) Move Platform back to holos.v1alpha1
No need to have a separate package for the PlatformService and related
protobuf messages.
2024-05-06 13:47:37 -07:00
Jeff McCune
adb8177026 Merge pull request #166 from holos-run/jeff/165-deploy-holos
(#165) Deploy Holos to Dev
2024-05-06 11:23:48 -07:00
Jeff McCune
4e8fa5abda (#165) Bump dev deployment to 0.73.1 2024-05-06 11:22:24 -07:00
Jeff McCune
6894f45b6c (#165) Deploy Holos to Dev
This patch deploys holos to the dev environment on the k2 cluster.  It's
accessible at https://app.dev.k2.holos.run/ behind the auth proxy by
default.
2024-05-06 11:10:29 -07:00
Jeff McCune
89d25be837 (#161) Wrap main router outlet in <main></main> 2024-05-06 09:16:15 -07:00
Jeff McCune
5b33e48552 (#161) Reasonably advanced form modeling the reference platform
This form goes a good way toward capturing what we need to configure the
entire reference platform.  Elements and sections are responsive to
which cloud providers are selected, which achieves my goal of modeling a
reasonably advanced form using only JSON data produced by CUE.

To write the form via the API:

    cue export ./forms/platform/ --out json \
      | jq '{platform_id: "'${platformId}'", fields: .spec.fields}' \
      | grpcurl -H "x-oidc-id-token: $(holos token)" -d @ ${host}:443 \
      holos.platform.v1alpha1.PlatformService.PutForm
2024-05-04 20:16:09 -07:00
Jeff McCune
79e8ab639a (#161) Fix the FormGroup & Refactor API
The way we were organizing fields into section broke Formly validation.
This patch fixes the problem by using the recommended approach of
[Nested Forms][1].

This patch also refactors the PlatformService API to clean it up.
GetForm / PutForm are separated from the Platform methods.  Similarly
GetModel / PutModel are separated out and are specific to get and put
the model data.

NOTE: I'm not sure we should have separated out the platform service
into it's own protobuf package.  Seems maybe unnecessary.

❯ grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"platform_id":"018f36fb-e3ff-7f7f-a5d1-7ca2bf499e94"}' jeff.app.dev.k2.holos.run:443 holos.platform.v1alpha1.PlatformService.GetModel
{
  "model": {
    "org": {
      "contactEmail": "platform@openinfrastructure.co",
      "displayName": "Open Infrastructure Services LLC",
      "domain": "ois.run",
      "name": "ois"
    },
    "privacy": {
      "country": "earth",
      "regions": [
        "us-east-2",
        "us-west-2"
      ]
    },
    "terms": {
      "didAgree": true
    }
  }
}

[1]: https://formly.dev/docs/examples/other/nested-formly-forms
2024-05-04 10:14:37 -07:00
Jeff McCune
a0cc673736 (#150) Wire up select and multi select boxes
This patch wires up a Select and a Multi Select box.  This patch also
establishes a decision as it relates to Formly TypeScript / gRPC Proto3
/ CUE definitions of the form data structure.  The decision is to use
gRPC as a transport for any JSON to avoid friction trying to fit Formly
types into Proto3 messages.

Note when using google.protobuf.Value messages with bufbuild/connect-es,
we need to round trip them one last time through JSON to get the
original JSON on the other side.  This is because connect-es preserves
the type discriminators in the case and value fields of the message.

Refer to: [Accessing oneof
groups](https://github.com/bufbuild/protobuf-es/blob/main/docs/runtime_api.md#accessing-oneof-groups)

NOTE: On the wire, carry any JSON as field configs for expedience.  I
attempted to reflect FormlyFieldConfig in protobuf, but it was too time
consuming.  The loosely defined Formly json data API creates significant
friction when joined with a well defined protobuf API.  Therefore, we do
not specify anything about the Forms API, convey any valid JSON, and
leave it up to CUE and Formly on the sending and receiving side of the
API.

We use CUE to define our own holos form elements as a subset of the loose
Formly definitions.  We further hope Formly will move toward a better JSON
data API, but it's unlikely.  Consider replacing Formly entirely and
building on top of the strongly typed Angular Dyanmic Forms API.

Refer to: https://github.com/ngx-formly/ngx-formly/blob/v6.3.0/src/core/src/lib/models/fieldconfig.ts#L15
Consider: https://angular.io/guide/dynamic-form

Usage:

Generate the form from CUE

    cue export ./forms/platform/ --out json | jq -cM | pbcopy

Store the form JSON in the config_values column of the platforms table.

View the form, and submit some data. Then get the data back out for use rendering the platform:

    grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"platform_id":"'${platformId}'"}' $holos holos.v1alpha1.PlatformService.GetConfig

```json
{
  "platform": {
    "spec": {
      "config": {
        "user": {
          "sections": {
            "org": {
              "fields": {
                "contactEmail": "jeff@openinfrastructure.co",
                "displayName": "Open Infrastructure Services LLC",
                "domain": "ois.run",
                "name": "ois"
              }
            },
            "privacy": {
              "fields": {
                "country": "earth",
                "regions": [
                  "us-east-2",
                  "us-west-2"
                ]
              }
            },
            "terms": {
              "fields": {
                "didAgree": true
              }
            }
          }
        }
      }
    }
  }
}
```
2024-05-03 10:42:03 -07:00
Jeff McCune
d06ecfadc8 (#150) Refactor PlatformService.GetConfig for use with CUE
Problem:
The GetConfig response value isn't directly usable with CUE without some
gymnastics.

Solution:
Refactor the protobuf definition and response output to make the user
defined and supplied config values provided by the API directly usable
in the CUE code that defines the platform.

Result:

The top level platform config is directly usable in the
`internal/platforms/bare` directory:

    grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"platform_id":"'${platformID}'"}' $host \
      holos.v1alpha1.PlatformService.GetConfig \
      > platform.holos.json

Vet the user supplied data:

    cue vet ./ -d '#PlatformConfig' platform.holos.json

Build the holos component.  The ConfigMap consumes the user supplied
data:

    cue export --out yaml -t cluster=k2 ./components/configmap platform.holos.json \
      | yq .spec.components

Note the data provided by the input form is embedded into the
ConfigMap managed by Holos:

```yaml
KubernetesObjectsList:
  - metadata:
      name: platform-configmap
    apiObjectMap:
      ConfigMap:
        platform: |
          metadata:
            name: platform
            namespace: default
            labels:
              app.holos.run/managed: "true"
          data:
            platform: |
              kind: Platform
              spec:
                config:
                  user:
                    sections:
                      org:
                        fields:
                          contactEmail: jeff@openinfrastructure.co
                          displayName: Open Infrastructure Services LLC
                          domain: ois.run
                          name: ois
              apiVersion: app.holos.run/v1alpha1
              metadata:
                name: bare
                labels: {}
                annotations: {}
              holos:
                flags:
                  cluster: k2
          kind: ConfigMap
          apiVersion: v1
    Skip: false
```
2024-05-02 06:39:33 -07:00
Jeff McCune
64a117b0c3 (#150) Add PlatformService.GetConfig and refactor ConfigValues proto
Problem:
The use of google.protobuf.Any was making it awkward to work with the
data provided by the user.  The structure of the form data is defined by
the platform engineer, so the intent of Any was to wrap the data in a
way we can pass over the network and persist in the database.

The escaped JSON encoding was problematic and error prone to decode on
the other end.

Solution:
Define the Platform values as a two level map with string keys, but with
protobuf message fields "sections" and "fields" respectively.  Use
google.protobuf.Value from the struct package to encode the actual
value.

Result:
In TypeScript, google.protobuf.Value encodes and decodes easily to a
JSON value.  On the go side, connect correctly handles the value as
well.

No more ugly error prone escaping:

```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"platform_id":"'${platformId}'"}' $host holos.v1alpha1.PlatformService.GetConfig
{
  "sections": {
    "org": {
      "fields": {
        "contactEmail": "jeff@openinfrastructure.co",
        "displayName": "Open Infrastructure Services LLC",
        "domain": "ois.run",
        "name": "ois"
      }
    }
  }
}
```

This return value is intended to be directly usable in the CUE code, so
we may further nest the values into a platform.spec key.
2024-05-01 21:30:30 -07:00
Jeff McCune
cf006be9cf (#150) Add SystemService DropTables and SeedDatabase
Makes it easier to reset the database and give Gary and Nate access to
the same organization I'm in so they can provide feedback.
2024-05-01 14:30:13 -07:00
Jeff McCune
45ad3d8e63 (#150) Fix 500 error when config values aren't provided
AddPlatform was failing with a 500 error trying to decode a nil byte
slice when adding a platform without providing any values.
2024-05-01 11:31:25 -07:00
Jeff McCune
441c968c4f (#150) Look up user by iss sub, not email.
Also log when orgs are created.
2024-05-01 10:02:08 -07:00
Jeff McCune
99f2763fdf (#150) Store Platform Config Form and Values as JSON
This patch changes the backend to store the platform config form
definition and the config values supplied by the form as JSON in the
database.

The gRPC API does not change with this patch, but may need to depending
on how this works and how easy it is to evolve the data model and add
features.
2024-05-01 09:11:53 -07:00
Jeff McCune
1312395a11 (#150) Fix platforms page links
The links were hard to click.  This makes the links a much larger click
target following the example at https://material.angular.io/components/list/overview#navigation-lists
2024-05-01 08:51:29 -07:00
Jeff McCune
615f147bcb (#150) Add PutPlatformConfig to store the config values
This patch is a work in progress wiring up the form to put the values to
the holos server using grpc.

In an effort to simplify the platform configuration, the structure is a
two level map with the top level being configuration sections and the
second level being the fields associated with the config section.

To support multiple kinds of values and field controls, the values are
serialized to JSON for rpc over the network and for storage in the
database.  When they values are used, either by the UI or by the `holos
render` command, they're to be unmarshalled and in-lined into the
Platform Config data structure.

Pick back up ensuring the Platform rpc handler correctly encodes and
decodes the structure to the database.

Consider changing the config_form and config_values fields to JSON field
types in the database.  It will likely make working with this a lot
easier.

With this patch we're ready to wire up the holos render command to fetch
the platform configuration and create the end to end demo.

Here's essentially what the render command will fetch and lay down as a
json file for CUE:

```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"platform_id":"018f2c4e-ecde-7bcb-8b89-27a99e6cc7a1"}' jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.GetPlatform | jq .platform.config.values
{
  "sections": {
    "org": {
      "values": {
        "contactEmail": "\"platform@openinfrastructure.co\"",
        "displayName": "\"Open Infrastructure Services  LLC\"",
        "domain": "\"ois.run\"",
        "name": "\"ois\""
      }
    }
  }
}
```
2024-04-30 20:21:15 -07:00
Jeff McCune
d0ad3bfc69 (#150) Add Platform Detail to edit platform config
This patch adds a /platform/:id route path to a PlatformDetail
component.  The platform detail component calls the GetPlatform method
given the platform ID and renders the platform config form on the detail
tab.

The submit button is not yet wired up.

The API for adding platforms changes, allowing raw json bytes using the
RawConfig.  The raw bytes are not presented on the read path though,
calling GetPlatforms provides the platform and the config form inline in
the response.

Use the `raw_config` field instead of `config` when creating the form
data.

```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" -d @ jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.AddPlatform <<EOF
{
  "platform": {
    "org_id": "018f27cd-e5ac-7f98-bfe1-2dbab208a48c",
    "name": "bare2",
    "raw_config": {
      "form": "$(cue export ./forms/platform/ --out json | jq -cM | base64 -w0)"
    }
  }
}
EOF
```
2024-04-30 14:02:49 -07:00
Jeff McCune
fe58a33747 (#150) Add holos.v1alpha1.PlatformService.GetForm
The GetForm method is intended for the Angular frontend to get
[FormlyFieldConfig][1] data for each section of the Platform config.

[1]: https://formly.dev/docs/api/core/#formlyfieldconfig

Steps to exercise for later testing:

Add the form definition to the database:

```
grpcurl -H "x-oidc-id-token: $(holos token)" -d @ jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.AddPlatform <<EOF
{
  "platform": {
    "org_id": "018f27cd-e5ac-7f98-bfe1-2dbab208a48c",
    "name": "bare${RANDOM}",
    "config": {
      "form": "$(cue export ./forms/platform/ --out json | jq -cM | base64 -w0)"
    }
  }
}
EOF
```

Get the form definition back out:

```

❯ grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"platform_id":"018f2bc1-6590-7670-958a-9f3bc02b658f"}' jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.GetForm
{
  "apiVersion": "forms.holos.run/v1alpha1",
  "kind": "PlatformForm",
  "metadata": {
    "name": "bare"
  },
  "spec": {
    "sections": [
      {
        "name": "org",
        "displayName": "Organization",
        "description": "Organization config values are used to derive more specific configuration values throughout the platform.",
        "fieldConfigs": [
          {
            "key": "name",
            "type": "input",
            "props": {
              "label": "Name",
              "placeholder": "example",
              "description": "DNS label, e.g. 'example'",
              "required": true
            }
          },
          {
            "key": "domain",
            "type": "input",
            "props": {
              "label": "Domain",
              "placeholder": "example.com",
              "description": "DNS domain, e.g. 'example.com'",
              "required": true
            }
          },
          {
            "key": "displayName",
            "type": "input",
            "props": {
              "label": "Display Name",
              "placeholder": "Example Organization",
              "description": "Display name, e.g. 'Example Organization'",
              "required": true
            }
          },
          {
            "key": "contactEmail",
            "type": "input",
            "props": {
              "label": "Contact Email",
              "placeholder": "platform-team@example.com",
              "description": "Technical contact email address",
              "required": true
            }
          }
        ]
      }
    ]
  }
}
```

References

```
❯ cue export ./forms/platform/ --out yaml | yq
apiVersion: forms.holos.run/v1alpha1
kind: PlatformForm
metadata:
  name: bare
spec:
  sections:
    - name: org
      displayName: Organization
      description: Organization config values are used to derive more specific configuration values throughout the platform.
      fieldConfigs:
        - key: name
          type: input
          props:
            label: Name
            placeholder: example
            description: DNS label, e.g. 'example'
            required: true
        - key: domain
          type: input
          props:
            label: Domain
            placeholder: example.com
            description: DNS domain, e.g. 'example.com'
            required: true
        - key: displayName
          type: input
          props:
            label: Display Name
            placeholder: Example Organization
            description: Display name, e.g. 'Example Organization'
            required: true
        - key: contactEmail
          type: input
          props:
            label: Contact Email
            placeholder: platform-team@example.com
            description: Technical contact email address
            required: true
```
2024-04-29 14:24:16 -07:00
Jeff McCune
26e537e768 (#150) Add platform config form, values, cue
This patch adds 4 fields to the Platform table:

 1. Config Form represents the JSON FormlyFieldConfig for the UI.
 2. Config CUE represents the CUE file containing a definition the
    Config Values must unify with.
 3. Config Definition is the CUE definition variable name used to unify
    the values with the cue code.  Should be #PlatformSpec in most
    cases.
 4. Config Values represents the JSON values provided by the UI.

The use case is the platform engineer defines the #PlatformSpec in cue,
and provides the form field config.  The platform engineer then provides
1-3 above when adding or updating a Platform.

The UI then presents the form to the end user and provides values for 4
when the user submits the form.

This patch also refactors the AddPlatform method to accept a Platform
message.  To do so we make the id field optional since it is server
assigned.

The patch also adds a database constraint to ensure platform names are
unique within the scope of an organization.

Results:

Note how the CUE representation of the Platform Form is exported to JSON
then converted to a base64 encoded string, which is the protobuf JSON
representation of a bytes[] value.

```
grpcurl -H "x-oidc-id-token: $(holos token)" -d @ jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.AddPlatform <<EOF
{
  "platform": {
    "id": "0d3dc0c0-bbc8-41f8-8c6e-75f0476509d6",
    "org_id": "018f27cd-e5ac-7f98-bfe1-2dbab208a48c",
    "name": "bare",
    "config": {
      "form": "$(cd internal/platforms/bare && cue export ./forms/platform/ --out json | jq -cM | base64 -w0)"
    }
  }
}
EOF
```

Note the requested platform ID is ignored.

```
{
  "platforms": [
    {
      "id": "018f2af9-f7ba-772a-9db6-f985ece8fed1",
      "timestamps": {
        "createdAt": "2024-04-29T17:49:36.058379Z",
        "updatedAt": "2024-04-29T17:49:36.058379Z"
      },
      "name": "bare",
      "creator": {
        "id": "018f27cd-e591-7f98-a9d2-416167282d37"
      },
      "config": {
        "form": "eyJhcGlWZXJzaW9uIjoiZm9ybXMuaG9sb3MucnVuL3YxYWxwaGExIiwia2luZCI6IlBsYXRmb3JtRm9ybSIsIm1ldGFkYXRhIjp7Im5hbWUiOiJiYXJlIn0sInNwZWMiOnsic2VjdGlvbnMiOlt7Im5hbWUiOiJvcmciLCJkaXNwbGF5TmFtZSI6Ik9yZ2FuaXphdGlvbiIsImRlc2NyaXB0aW9uIjoiT3JnYW5pemF0aW9uIGNvbmZpZyB2YWx1ZXMgYXJlIHVzZWQgdG8gZGVyaXZlIG1vcmUgc3BlY2lmaWMgY29uZmlndXJhdGlvbiB2YWx1ZXMgdGhyb3VnaG91dCB0aGUgcGxhdGZvcm0uIiwiZmllbGRDb25maWdzIjpbeyJrZXkiOiJuYW1lIiwidHlwZSI6ImlucHV0IiwicHJvcHMiOnsibGFiZWwiOiJOYW1lIiwicGxhY2Vob2xkZXIiOiJleGFtcGxlIiwiZGVzY3JpcHRpb24iOiJETlMgbGFiZWwsIGUuZy4gJ2V4YW1wbGUnIiwicmVxdWlyZWQiOnRydWV9fSx7ImtleSI6ImRvbWFpbiIsInR5cGUiOiJpbnB1dCIsInByb3BzIjp7ImxhYmVsIjoiRG9tYWluIiwicGxhY2Vob2xkZXIiOiJleGFtcGxlLmNvbSIsImRlc2NyaXB0aW9uIjoiRE5TIGRvbWFpbiwgZS5nLiAnZXhhbXBsZS5jb20nIiwicmVxdWlyZWQiOnRydWV9fSx7ImtleSI6ImRpc3BsYXlOYW1lIiwidHlwZSI6ImlucHV0IiwicHJvcHMiOnsibGFiZWwiOiJEaXNwbGF5IE5hbWUiLCJwbGFjZWhvbGRlciI6IkV4YW1wbGUgT3JnYW5pemF0aW9uIiwiZGVzY3JpcHRpb24iOiJEaXNwbGF5IG5hbWUsIGUuZy4gJ0V4YW1wbGUgT3JnYW5pemF0aW9uJyIsInJlcXVpcmVkIjp0cnVlfX0seyJrZXkiOiJjb250YWN0RW1haWwiLCJ0eXBlIjoiaW5wdXQiLCJwcm9wcyI6eyJsYWJlbCI6IkNvbnRhY3QgRW1haWwiLCJwbGFjZWhvbGRlciI6InBsYXRmb3JtLXRlYW1AZXhhbXBsZS5jb20iLCJkZXNjcmlwdGlvbiI6IlRlY2huaWNhbCBjb250YWN0IGVtYWlsIGFkZHJlc3MiLCJyZXF1aXJlZCI6dHJ1ZX19XX1dfX0K"
      }
    }
  ]
}
```
2024-04-29 10:53:23 -07:00
Jeff McCune
ad70a6c4fe (#150) Add holos.v1alpha1.PlatformService.AddPlatform
This patch adds a basic AddPlatform method that adds a platform with a
name and a display name.

Next steps are to add fields for the Platform Config Form definition and
the Platform Config values submitted from the form.
2024-04-29 09:35:49 -07:00
Jeff McCune
22a04da6bb (#150) Add holos.v1alpha1.PlatformService.GetPlatforms
Next step: AddPlatform

Also consider extracting the queries to get the requested org_id to a
helper function.  This will likely eventually move to an interceptor
because every request is org scoped and needs authorization checks
against the org.

```
grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"org_id":"018f27cd-e5ac-7f98-bfe1-2dbab208a48c"}' jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.GetPlatforms
```
2024-04-28 20:21:32 -07:00
Jeff McCune
dc97fe0ff0 (#150) Define a PlatformForm for platform design
Problem:
Platform engineers need the ability to define custom input fields for
their own platform level configuration values.  The holos web UI needs
to present the platform config values in a clean way.  The values
entered on the form need to make their way into the top level
Platform.spec field for use across all components and clusters in the
platform.

Solution:
Define a Platform Form in a forms cue package.  The output of this
definition is intended to be sent to the holos server to provide to the
web UI.

Result:
Platform engineers can define their platform config input values in
their infrastructure repository.  For example, the bare platform form
inputs are defined at `platforms/bare/forms/platform/platform-form.cue`.

This cue file produces [FormlyFieldConfig][1] output.

```console
cue export ./forms/platform/ --out yaml
```

```yaml
apiVersion: forms.holos.run/v1alpha1
kind: PlatformForm
metadata:
  name: bare
spec:
  sections:
    - name: org
      displayName: Organization
      description: Organization config values are used to derive more specific configuration values throughout the platform.
      fieldConfigs:
        - key: name
          type: input
          props:
            label: Name
            placeholder: example
            description: DNS label, e.g. 'example'
            required: true
        - key: domain
          type: input
          props:
            label: Domain
            placeholder: example.com
            description: DNS domain, e.g. 'example.com'
            required: true
        - key: displayName
          type: input
          props:
            label: Display Name
            placeholder: Example Organization
            description: Display name, e.g. 'Example Organization'
            required: true
        - key: contactEmail
          type: input
          props:
            label: Contact Email
            placeholder: platform-team@example.com
            description: Technical contact email address
            required: true
```

Next Steps:
Add a holos subcommand to produce the output and store it in the
backend.  Wire the front end to fetch the form config from the backend.

[1]: https://formly.dev/docs/api/core#formlyfieldconfig
2024-04-28 11:25:06 -07:00
Jeff McCune
9ca97c6e01 Merge pull request #148 from holos-run/jeff/147-cue-oom
(#147) Add holos render --print-instances flag
2024-04-26 16:31:14 -07:00
Jeff McCune
924653e240 (#150) Bare Platform
This patch adds a bare platform that does nothing but render a configmap
containing the platform config structure itself.

The definition of the platform structure is firming up.  The platform
designer, which may be a holos customer, is responsible for defining the
structure of the `platform.spec` output field.

Us holos developers have a reserved namespace to add configuration
fields and data in the `platform.holos` output file.

Beyond these two fields, the platform config structure has TypeMeta and
ObjectMeta fields similar to a kubernetes api object to support
versioning the platform config data, naming the platform, annotating the
platform, and labeling the platform.

The path forward from here is to:

 1. Eventually move the stable definitions into a CUE module that gets
    imported into the user's package.
 2. As a platform designer, add the organization field to the
    #PlatformSpec definition as a CUE definition.
 3. As a platform designer, add the organization field Form data
    structure as a JSON file.
 4. Add an API to upload the #PlatformSpec cue file and the
    #PlatformSpec form json file to the saas backend.
 5. Wire up Angular to pull the form json from the API and render the
    form.
 6. Wire up Angular to write the form data to a gRPC service method.
 7. Wire up the `holos cli` to read the form data from a gRPC service
    method.
 8. Tie it all together where the holos cli renders the configmap.
2024-04-26 16:14:30 -07:00
Jeff McCune
59d48f8599 (#146) Platform Config Mock Up
This patch adds a mock up of the platform config.  The goal is to use
this to connect to an anemic example platform built from `holos init`.
2024-04-26 11:29:58 -07:00
Jeff McCune
90f8eab816 (#144) Tidy go.mod and package.json 2024-04-25 19:14:20 -07:00
Jeff McCune
9ae45e260d (#147) Add holos render --print-instances flag
To enumerate all of the instances that could be run in separate
processes with xargs instead of run in the for loop in the Builder Run
method.
2024-04-25 13:59:10 -07:00
Jeff McCune
aee15f95e2 Merge pull request #145 from holos-run/jeff/144-organization-selector
(#144) Profile Button
2024-04-25 09:55:55 -07:00
Jeff McCune
1c540ac375 (#144) Profile Button and Organization Selector
This patch adds an organization "selector" that's really just a place
holder.  The active organization is the last element in the list
returned by the GetCallerOrganizations method for now.

The purpose is to make sure we have the structure in place for more than
one organizations without needing to implement full support for the
feature at this early stage.

The Angular frontend is expected to call the activeOrg() method of the
OrganizationService.  In the future this could store the state of which
organization the user has selected.  The purpose is to return an org id
to send as a request parameter for other requests.

Note this patch also implements refresh behavior.  The list of orgs is
fetched once on application load.  If there is no user, or the user has
zero orgs, the user is created and an organization is added with them as
an owner.  This is accompished using observable pipes.

The pipe is tied to a refresh behavior.  Clicking the org button
triggers the refresh behavior, which executes the pipe again and
notifies all subscribers.

This works quite well and should be idiomatic angular / rxjs.  Clicking
the button automatically updates the UI after making the necessary API
calls.
2024-04-25 09:55:13 -07:00
Jeff McCune
5b0e883ac9 (#144) Get or Create the orgranization
This patch adds the OrganizationService to the Angular front end and
displays a simple list of the organizations the user is a member of in
the profile card.

There isn't a service yet to return the currently selected
organization, but that could be a simple method to return the most
recent entry in the list until we put something more complicated in
place like local storage of what the user has selected.

It may make sense to put a database constraint on the number of
organizations until we implement the feature later, it's too early to do
so now, I just want to make sure it's possible to add later.
2024-04-25 07:02:17 -07:00
Jeff McCune
9a2519af71 (#144) Make the linter happy 2024-04-24 13:41:45 -07:00
Jeff McCune
9b9ff601c0 (#144) Call GetCallerClaims once instead of multiple times
Problem:
When loading the page the GetCallerClaims rpc method is called multiple
times unnecessarily.

Solution:
Use [shareReplay][1] to replay the last observable event for all
subscribers, including subscribers coming late to the party.

Result:
Network inspector in chrome indicates GetCallerClaims is called once and
only once.

[1]: https://rxjs.dev/api/operators/shareReplay
2024-04-24 12:44:44 -07:00
Jeff McCune
2f798296dc (#144) Profile Button
This patch adds a ProfileButton component which makes a ConnectRPC gRPC
call to the `holos.v1alpha1.UserService.GetCallerClaims` method and
renders the profile button based on the claims.

Note, in the network inspector there are two API calls to
`holos.v1alpha1.UserService.GetCallerClaims` which is unfortunate.  A
follow up patch might be good to fix this.
2024-04-24 12:23:54 -07:00
Jeff McCune
2b2ff63cad (#144) Connect /ui to ng serve for hot reload
Problem:
It's slow to build the angular app, compile it into the go executable,
copy it to the pod, then restart the server.

Solution:
Configure the mesh to route /ui to `ng serve` running on my local
host.

Result:
Navigating to https://jeff.app.dev.k2.holos.run/ui gets responses from
the ng development server.

Use:

    ng serve --host 0.0.0.0
2024-04-23 20:30:02 -07:00
Jeff McCune
3b135c09f3 (#144) Make a ConnectRPC call to the GetUserClaims method
This patch wires up an Angular RxJS Observable to the result of a gRPC
call to the `holos.v1alpha1.UserService.GetCallerClaims` method.

The implementation is a combination of [this connect example][1] and the
official [angular data][2] guide.

[1]: https://github.com/connectrpc/examples-es/tree/main/angular
[2]: https://angular.io/start/start-data#configuring-the-shippingcomponent-to-use-cartservice
2024-04-23 17:18:35 -07:00
Jeff McCune
28813eba5b (#126) v0.69.0 2024-04-23 10:24:58 -07:00
Jeff McCune
02ff765f54 Merge pull request #143 from holos-run/jeff/126-registration
(#126) Minimal API to register users and organizations
2024-04-23 09:00:07 -07:00
Jeff McCune
fe8a806132 (#126) Refactor to GetCallerX / CreateCallerX
This patch simplifies the user and organization registration and query
for the UI.  The pattern clients are expected to follow is to create if
the get fails.  For example, the following pseudo-go-code is the
expected calling convention:

    var entity *ent.User
    entity, err := Get()
    if err != nil {
      if ent.MaskNotFound(err) == nil {
        entity = Create()
      } else {
        return err
      }
    }
    return entity

This patch adds the following service methods.  For initial
registration, all input data comes from the id token claims of the
authenticated user.

```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 list | xargs -n1 grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 list
holos.v1alpha1.OrganizationService.CreateCallerOrganization
holos.v1alpha1.OrganizationService.GetCallerOrganizations
holos.v1alpha1.UserService.CreateCallerUser
holos.v1alpha1.UserService.GetCallerClaims
holos.v1alpha1.UserService.GetCallerUser
```
2024-04-23 08:43:23 -07:00
Jeff McCune
6626d58301 (#126) Add OrganizationService
Next step after this is to simplify the calling convention to a get
followed by a create if the get fails.
2024-04-23 05:28:07 -07:00
Jeff McCune
cb0911e890 (#126) Add holos.v1alpha1.UserService.GetUser
❯ grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 holos.v1alpha1.UserService.GetUser
{
  "user": {
    "id": "018f07f4-4f9c-7b69-9d9e-07bf7bb4fe33",
    "email": "jeff@openinfrastructure.co",
    "name": "Jeff McCune",
    "timestamps": {
      "createdAt": "2024-04-22T22:36:42.780492Z",
      "updatedAt": "2024-04-22T22:36:42.780492Z"
    }
  }
}
2024-04-22 16:49:25 -07:00
Jeff McCune
3745a68dc5 (#126) Add unique index on user iss sub fields
The server will frequently look up the user record given the iss and sub
claims from the id token, index them and make sure the combination of
the two is unique.
2024-04-22 16:14:40 -07:00
Jeff McCune
fd64830476 (#126) Rename HolosService to UserService
Organize services by the resource they manage.
2024-04-22 16:05:32 -07:00
Jeff McCune
1ee0fa9c1f (#126) User Registration via API
With this patch user registration works with grpcurl.  Nothing in the
web UI yet.

```
grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 holos.v1alpha1.HolosService.RegisterUser
```

Cannot register twice:

```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 holos.v1alpha1.HolosService.RegisterUser
ERROR:
  Code: FailedPrecondition
  Message: user.go:26: ent: constraint failed: ERROR: duplicate key value violates unique constraint "users_email_key" (SQLSTATE 23505)
```

GetUserClaims works though:

```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 holos.v1alpha1.HolosService.GetUserClaims
{
  "iss": "https://login.ois.run",
  "sub": "261773693724656988",
  "email": "jeff@openinfrastructure.co",
  "emailVerified": true,
  "name": "Jeff McCune"
}
```
2024-04-22 15:38:07 -07:00
Jeff McCune
8fab325b0a (#126) Add gRPC reflection
So grpcurl works as expected:

```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 list
holos.v1alpha1.HolosService
```
2024-04-22 15:02:08 -07:00
Jeff McCune
858ffad913 (#126) Add holos token command for grpcurl 2024-04-22 14:54:09 -07:00
Jeff McCune
62735b99e7 (#126) Update Tiltfile to use holos.run for dev
This patch updates the Tiltfile to use the holos.run domain which is
integrated with the default Gateway.
2024-04-22 13:42:18 -07:00
Jeff McCune
29ab9c6300 (#141) Install provisioner helper.rb from entrypoint
And add a script to reset the choria provisioner credentials and config.
2024-04-22 13:20:38 -07:00
Jeff McCune
debc01c7de (#141) Fix Incorrect Provisioning Token foo given
The `make-provisioner-jwt` incorrectly used the choria broker password
as the provisioning token.  In the reference [setup.sh][1] both the
token and the `broker_provisioning_password` are set to `s3cret` so I
confused the two, but they are actually different values.

This patch ensures the provisioning token configured in
`provisioner.yaml` matches the token embedded into the provisioning.jwt
file using `choria jwt provisioning` via the `make-provisioner-jwt`
script.

[1]: 6dbc8fd105/example/setup/templates/provisioner/provisioner.yaml (L6)
2024-04-22 12:31:10 -07:00
Jeff McCune
c07f35ecd6 (#141) Fix holos controller invalid websocket connection error
Problem:
When the ingress default Gateway AuthorizationPolicy/authpolicy-custom
rule is in place the choria machine room holos controller fails to
connect to the provisioner broker with the following error:

```
❯ holos controller run --config=agent.cfg
WARN[0000] Starting controller version 0.68.1 with config file /home/jeff/workspace/holos-run/holos/hack/choria/agent/agent.cfg  leader=false
WARN[0000] Switching to provisioning configuration due to build defaults and missing /home/jeff/workspace/holos-run/holos/hack/choria/agent/agent.cfg
WARN[0000] Setting anonymous TLS mode during provisioning  component=server connection=coffee.home identity=coffee.home
WARN[0000] Initial connection to the Broker failed on try 1: invalid websocket connection  component=server connection=coffee.home identity=coffee.home
WARN[0000] Initial connection to the Broker failed on try 2: invalid websocket connection  component=server connection=coffee.home identity=coffee.home
WARN[0002] Initial connection to the Broker failed on try 3: invalid websocket connection  component=server connection=coffee.home identity=coffee.home
```

This problem is caused because the provisioning token url is set to
`wss://jeff.provision.dev.k2.holos.run:443` which has the port number
specified.

Solution:
Follow the upstream istio guidance of [Writing Host Match Policies][1]
to match host headers with or without the port specified.

Result:
The controller is able to connect to the provisioner broker:

[1]: https://istio.io/latest/docs/ops/best-practices/security/#writing-host-match-policies
2024-04-22 12:31:10 -07:00
Jeff McCune
c8f528700c (#141) Fix error: do not know how to handle choria_provisioning purpose token
Solution:
remove the plugin.security.choria.ca setting
2024-04-22 12:30:16 -07:00
Jeff McCune
896248c237 (#141) Try and connect holos controller to the provisioner
Running into error:

time="2024-04-20T03:23:19Z" level=warning msg="Denying connection: verified error: do not know how to handle choria_provisioning purpose token, unverified error: <nil>" component=authentication remote="10.244.1.51:56338" stage=check
time="2024-04-20T03:23:19Z" level=error msg="192.168.2.21/10.244.1.51:56338 - wid:367 - authentication error" component=network_broker
2024-04-22 12:29:56 -07:00
Jeff McCune
74a181db21 (#133) Add missing Choria Provisioner deployment 2024-04-19 15:14:31 -07:00
Jeff McCune
ba10113342 (#133) Fix tls error when connecting to provisioner websocket
This problem fixes an error where the istio ingress gateway proxy failed
to verify the TLS certificate presented by the choria broker upstream
server.

    kubectl logs choria-broker-0

    level=error msg="websocket: TLS handshake error from 10.244.1.190:36142: remote error: tls: unknown certificate\n"

Istio ingress logs:

    kubectl -n istio-ingress logs -l app=istio-ingressgateway -f | grep --line-buffered '^{' | jq .

    "upstream_transport_failure_reason": "TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end"

Client curl output:

    curl https://jeff.provision.dev.k2.holos.run

    upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_i
nternal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

Explanation of error:

Istio defaults to expecting a tls certificate matching the downstream
host/authority which isn't how we've configured Choria.

Refer to [ClientTLSSettings][1]

> A list of alternate names to verify the subject identity in the
> certificate. If specified, the proxy will verify that the server
> certificate’s subject alt name matches one of the specified values. If
> specified, this list overrides the value of subject_alt_names from the
> ServiceEntry. If unspecified, automatic validation of upstream presented
> certificate for new upstream connections will be done based on the
> downstream HTTP host/authority header, provided
> VERIFY_CERTIFICATE_AT_CLIENT and ENABLE_AUTO_SNI environmental variables
> are set to true.

[1]: https://istio.io/latest/docs/reference/config/networking/destination-rule/#ClientTLSSettings
2024-04-19 13:13:09 -07:00
Jeff McCune
eb0207c92e (#133) Choria Provisioner
This patch is a work in progress to configure the provisioner to connect
to the broker.  Services and deployments are prefixed with choria for
clarity.
2024-04-19 13:13:08 -07:00
Jeff McCune
0fbcee8119 (#133) Extend the life of the Platform Issuer CA
The platform issuer root CA was set to expire after 90 days, the default
value.  This is too short.

Extend the life of the root CA beyond 100 years.
2024-04-19 11:29:17 -07:00
Jeff McCune
ce8bc798f6 (#133) Exclude nats and provision hosts from the auth proxy
Problem:
The identity aware auth proxy attached to the default gateway is
blocking access to NATS and the Choria Provisioner cluster.

Solution:
Add configuration that causes the project hosts to get added to the
exclusion list of the AuthorizationPolicy/authproxy-custom rule.

Result:
Requests bypass the auth proxy and go straight to the backend.  The
rules look like:

    kubectl get authorizationpolicy authproxy-custom -o yaml

```yaml
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: authproxy-custom
  namespace: istio-ingress
  labels:
    app.kubernetes.io/name: authproxy-custom
    app.kubernetes.io/part-of: istio-ingressgateway
spec:
  action: CUSTOM
  provider:
    name: ingressauth
  rules:
  - to:
    - operation:
        notHosts:
        - login.ois.run
        - vault.core.ois.run
        - provision.holos.run
        - nats.holos.run
        - provision.dev.holos.run
        - nats.dev.holos.run
        - jeff.provision.dev.holos.run
        - jeff.nats.dev.holos.run
        - gary.provision.dev.holos.run
        - gary.nats.dev.holos.run
        - nate.provision.dev.holos.run
        - nate.nats.dev.holos.run
        - provision.k2.holos.run
        - nats.k2.holos.run
        - provision.dev.k2.holos.run
        - nats.dev.k2.holos.run
        - jeff.provision.dev.k2.holos.run
        - jeff.nats.dev.k2.holos.run
        - gary.provision.dev.k2.holos.run
        - gary.nats.dev.k2.holos.run
        - nate.provision.dev.k2.holos.run
        - nate.nats.dev.k2.holos.run
    when:
    - key: request.headers[x-oidc-id-token]
      notValues:
      - '*'
  selector:
    matchLabels:
      istio: ingressgateway
```
2024-04-19 06:32:11 -07:00
Jeff McCune
996195d651 (#137) Fix ArgoCD PKCE login small comment 2024-04-18 14:39:13 -07:00
Jeff McCune
f00b29d3a3 (#137) Fix ArgoCD PKCE login
This patch configures ArgoCD to log in via PKCE.

Note the changes are primarily in platform.site.cue and ensuring the
emailDomain is set properly.  Note too the redirect URL needs to be
`/pkce/verify` when PKCE is enabled.  Finally, if the setting is
reconfigured make sure to clear cookies otherwise the incorrect
`/auth/callback` path may be used.
2024-04-18 14:37:06 -07:00
Jeff McCune
a6756ecf11 (#132) Update go deps to fix Machine Room 2024-04-18 13:35:58 -07:00
Jeff McCune
ef7ec30037 (#132) Use machine-room main branch
Our patch to use Args got merged.
2024-04-18 11:38:51 -07:00
Jeff McCune
1642787825 (#101) Fix port names in servers must be unique: duplicate name https
Problem:
Port names in the default Gateway.spec.servers.port field must be unique
across all servers associated with the workload.

Solution:
Append the fully qualified domain name with dots replaced with hyphens.

Result:
Port name is unique.
2024-04-18 11:21:05 -07:00
Jeff McCune
f83781480f (#101) Do not add Gateway.spec.servers for other clusters
Problem:
The default gateway in one cluster gets server entries for all hosts in
the problem.  This makes the list unnecessarily large with entries for
clusters that should not be handled on the current cluster.

For example, the k2 cluster has gateway entries to route hosts for k1,
k3, k4, k5, etc...

Solution:
Add a field to the CertInfo definition representing which clusters the
host is valid on.

Result:
Hosts which are valid on all clusters, e.g. login.ois.run, have all
project clusters added to the clusters field of the CertInfo.  Hosts
which are valid on a single cluster have the coresponding single entry
added.

When building resources, holos components should check if `#ClusterName`
is a valid field of the CertInfo.clusters field.  If so, the host is
valid for the current cluster.  If not, the host should be omitted from
the current cluster.
2024-04-18 11:10:16 -07:00
Jeff McCune
9b70205855 (#101) Use letsencrypt production instead of staging
Certificates issue OK from staging, switching to production.
2024-04-18 10:36:28 -07:00
Jeff McCune
0e4bf3c144 (#101) Manage certs on all clusters
We're going to do a big re-issue so might as well do it once so we don't
have to re-issue again to add more clusters to existing projects.
2024-04-18 10:16:55 -07:00
Jeff McCune
1241c74b41 (#101) Do not add the project name as a project host
Doing so forces unnecessary hosts for some projects.  For example,
iam.ois.run is useless for the iam project, the primary project host is
login to build login.ois.run.

Some projects may not need any hosts as well.

Better to let the user specify `project: foo: hosts: foo: _` if they
want it.
2024-04-18 09:59:21 -07:00
Jeff McCune
44fea098de (#101) Manage an ExternalSecret for every Server in the default Gateway
This patch loops over every Gateway.spec.servers entry in the default
gateway and manages an ExternalSecret to sync the credential from the
provisioner cluster.
2024-04-18 09:53:39 -07:00
Jeff McCune
52286efa25 (#101) Fix duplicate certs in holos components
Problem:
A Holos Component is created for each project stage, but all hosts for
all stages in the project are added.  This creates duplicates.

Solution:
Sort project hosts by their stage and map the holos component for a
stage to the hosts for that stage.

Result:
Duplicates are eliminated, the prod certs are not in the dev holos
component and vice-versa.
2024-04-18 09:17:49 -07:00
Jeff McCune
a1b2179442 (#101) Remove holos-saas-certs holos component
No longer needed now that project host certs are using wildcards and
organized nicely.
2024-04-18 06:32:06 -07:00
Jeff McCune
cffc430738 (#101) Provision wildcard certs for all Gateway servers
This patch provisions wildcard certs in the provisioning cluster.  The
CN matches the project stage host global hostname without any cluster
qualifiers.

The use of a wildcard in place of the environment name dns segment at
the leftmost position of the fully qualified dns name enables additional
environments to be configured without reissuing certificates.

This is to avoid the 100 name per cert limit in LetsEncrypt.
2024-04-18 06:26:29 -07:00
Jeff McCune
d76454272b (#101) Simplify the GatewayServers struct
Mapping each project host fqdn to the stage is unnecessary.  The list of
gateway servers is constructed from each FQDN in the project.

This patch removes the unnecessary struct mappings.
2024-04-18 05:32:19 -07:00
Jeff McCune
9d1e77c00f (#101) Define #ProjectHosts to manage project hosts
Problem:
It's difficult to map and reduce the collection of project hosts when
configuring related Certificate, Gateway.spec.servers, VirtualService,
and auth proxy cookie domain settings.

Solution:
Define #ProjectHosts which takes a project and provides Hosts which is a
struct with a fqdn key and a #CertInfo value.  The #CertInfo definition
is intended to provide everything need to reduce the Hosts property to
structs usful for the problematic resources mentioned previously.

Result:
Gateway.spec.servers are mapped using #ProjectHosts

Next step is to map the Certificate resources on the provisioner
cluster.
2024-04-17 21:59:04 -07:00
Jeff McCune
2050abdc6c (#101) Add wildcard support to project certs
Problem:
Adding environments to a project causes certs to be re-issued.

Solution:
Enable wildcard certs for per-environment namespaces like jeff, gary,
nate, etc...

Result:
Environments can be added to a project stage without needing the cert to
be re-issued.
2024-04-17 12:32:44 -07:00
Jeff McCune
3ea013c503 (#101) Consolidate certificates by project stage
This patch avoids LetsEncrypt rate limits by consolidating multiple dns
names into one certificate.

For each project host, create a certificate for each stage in the
project.  The certificate contains the dns names for all clusters and
environments associated with that stage and host.

This can become quite a list, the limit is 100 dnsNames.

For the Holos project which has 7 clusters and 4 dev environments, the
number of dns names is 32 (4 envs + 4 envs * 7 clusters = 32 dns names).

Still, a much needed improvement because we're limited to 50 certs per
week.

It may be worth considering wildcards for the per-developer
environments, which are the ones we'll likely spin up the most
frequently.
2024-04-17 11:58:46 -07:00
Jeff McCune
309db96138 (#133) Choria Broker for Holos Controller provisioning
This patch is a partial step toward getting the choria broker up
and running in my own namespace.  The choria broker is necessary for
provisioning machine room agents such as the holos controller.
2024-04-17 08:48:31 -07:00
Jeff McCune
283b4be71c (#132) Use forked version of machine-room
Until https://github.com/choria-io/machine-room/pull/12 gets merged
2024-04-16 19:46:36 -07:00
Jeff McCune
ab9bca0750 (#132) Controller Subcommand
This patch adds an initial holos controller subcommand.  The machine
room agent starts, but doesn't yet provision because we haven't deployed
the provisioning infrastructure yet.
2024-04-16 15:40:25 -07:00
Jeff McCune
ac2be67c3c (#130) NATS deployment with operator jwt
Configure NATS in a 3 Node deployment with resolver authentication using
an Operator JWT.

The operator secret nkeys are stored in the provisioner cluster.  Get
them with:

    holos get secret -n jeff-holos nats-nsc --print-key nsc.tgz | tar -tvzf-
2024-04-15 17:02:18 -07:00
Jeff McCune
6ffafb8cca (#127) Setup Routing using Dashboard Schematic
This patch sets up basic routing and a 404 not found page.  The Home and
Clusters page are generated from the [dashboard schematic][1]

    ng generate @angular/material:dashboard home
    ng generate @angular/material:dashboard cluster-list
    ng g c error-not-found

[1]: https://material.angular.io/guide/schematics#dashboard-schematic
2024-04-15 13:48:00 -07:00
Jeff McCune
590e6b556c (#127) Generate Angular Material navigation
Instead of trying to hand-craft a navigation sidebar and toolbar from
Youtube videos, use the [navigation schematic][1] to quickly get a "good
enough" UI.

    ng generate @angular/material:navigation nav

[1]: https://material.angular.io/guide/schematics#navigation-schematic
2024-04-15 10:43:24 -07:00
Jeff McCune
5dc5c6fbdf (#127) ng add @angular/material
And start working on the sidenav and toolbar.
2024-04-14 07:03:45 -07:00
Jeff McCune
cd8c9f2c32 (#127) ConnectRPC generated code 2024-04-13 11:03:19 -07:00
Jeff McCune
3490941d4c (#127) Frontend deps from make tools
Needed to generate the connectrpc bindings and build the holos
executable.
2024-04-12 20:09:41 -07:00
Jeff McCune
3f201df0c2 (#126) Configure Angular to align with frontend.go
Angular must build output into a path compatible with the Go
http.FileServer.  We cannot easily graft an fs.FS onto a sub-path, so we
need the `./ui/` path in the output.  This requires special
configuration from the Angular default application builder behavior.
2024-04-12 20:08:37 -07:00
Jeff McCune
4c22d515bd (#127) ng new holos
ng new holos --routing --skip-git --standalone
SCSS
No SSR
2024-04-12 20:07:17 -07:00
Jeff McCune
ec0ef1c4b3 (#127) Angular - Restart again
Restart again this time with SCSS instead of CSS.
2024-04-12 20:03:45 -07:00
Jeff McCune
1e51e2d49a (#127) Angular Navigation schematic
Following [Navigation schematic][1].

    ng generate @angular/material:navigation navigation

[1]: https://material.angular.io/guide/schematics#navigation-schematic
2024-04-12 19:45:26 -07:00
Jeff McCune
5186499b90 Revert "(#127) Angular - ng add ng-matero"
This reverts commit fc275e4164.

Yuck, don't like it.
2024-04-12 17:21:26 -07:00
Jeff McCune
fc275e4164 (#127) Angular - ng add ng-matero
Trying [ng-matero][1].  Seems to exceed the max prod budget of 1mb, but
worth trying anyway.

[1]: https://github.com/ng-matero/ng-matero
2024-04-12 17:18:33 -07:00
Jeff McCune
9fa466f7cf (#126) Build the front end app when building holos
Always build the front end app bundle when rebuilding the holos cli so
we're sure things are up to date.
2024-04-12 17:04:41 -07:00
Jeff McCune
efd6f256a5 (#126) Connect generated bindings for the frontend 2024-04-12 16:57:30 -07:00
Jeff McCune
f7f9d6b5f0 (#126) Angular Material - ng add @angular/material 2024-04-12 16:57:15 -07:00
Jeff McCune
0526062ab2 (#126) Configure Angular to align with frontend.go
Angular must build output into a path compatible with the Go
http.FileServer.  We cannot easily graft an fs.FS onto a sub-path, so we
need the `./ui/` path in the output.  This requires special
configuration from the Angular default application builder behavior.
2024-04-12 16:57:15 -07:00
Jeff McCune
a1ededa722 (#126) http.FileServer serves /ui instead of /app
This fixes Angular not being served up correctly.

Note, special configuration in Angular is necessary to get the build
output into the ui/ directory.  Refer to: [Output path configuration][1]
and [browser directory created in outputPath][2].

[1]: https://angular.io/guide/workspace-config#output-path-configuration
[2]: https://github.com/angular/angular-cli/issues/26304
2024-04-12 16:51:45 -07:00
Jeff McCune
9b09a02912 (#115) Angular new project with defaults
Setup angular with the defaults.  CSS, No SSR / Static Site Generation.

    npm install -g @angular/cli
    ng new holos

```
? Which stylesheet format would you like to use? CSS             [ https://developer.mozilla.org/docs/Web/CSS                     ]
? Do you want to enable Server-Side Rendering (SSR) and Static Site Generation (SSG/Prerendering)? No
```

```
CREATE holos/README.md (1059 bytes)
CREATE holos/.editorconfig (274 bytes)
CREATE holos/.gitignore (548 bytes)
CREATE holos/angular.json (2587 bytes)
CREATE holos/package.json (1036 bytes)
CREATE holos/tsconfig.json (857 bytes)
CREATE holos/tsconfig.app.json (263 bytes)
CREATE holos/tsconfig.spec.json (273 bytes)
CREATE holos/.vscode/extensions.json (130 bytes)
CREATE holos/.vscode/launch.json (470 bytes)
CREATE holos/.vscode/tasks.json (938 bytes)
CREATE holos/src/main.ts (250 bytes)
CREATE holos/src/favicon.ico (15086 bytes)
CREATE holos/src/index.html (291 bytes)
CREATE holos/src/styles.css (80 bytes)
CREATE holos/src/app/app.component.css (0 bytes)
CREATE holos/src/app/app.component.html (19903 bytes)
CREATE holos/src/app/app.component.spec.ts (913 bytes)
CREATE holos/src/app/app.component.ts (301 bytes)
CREATE holos/src/app/app.config.ts (227 bytes)
CREATE holos/src/app/app.routes.ts (77 bytes)
CREATE holos/src/assets/.gitkeep (0 bytes)
✔ Packages installed successfully.
```
2024-04-12 15:07:38 -07:00
Jeff McCune
657a5e82a5 (#115) Remove Angular SSR
We don't want Angular Server Side Rendering, we want plain old client
side angular.
2024-04-12 14:57:39 -07:00
Jeff McCune
1eece02254 (#126) Angular Material UI
ng add @angular/material

```
❯ ng add @angular/material
Skipping installation: Package already installed
? Choose a prebuilt theme name, or "custom" for a custom theme: Indigo/Pink        [ Preview: https://material.angular.io?theme=indigo-pink ]
? Set up global Angular Material typography styles? Yes
? Include the Angular animations module? Include and enable animations Yes
```
2024-04-12 14:16:45 -07:00
Jeff McCune
c866b47dcb (#126) Check for errors decoding claims
Return an empty claims struct when there's an error.
2024-04-12 14:16:44 -07:00
Jeff McCune
ff52ec750b (#126) Try to fix golangci-lint
It's doing way too much, might want to consider something else.

Getting these errors:

```
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.dockerignore: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.envrc: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.gitattributes: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.github/CODEOWNERS: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.github/buf-logo.svg: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.github/dependabot.yml: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.github/workflows/add-to-project.yaml: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.github/workflows/back-to-development.yaml: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.github/workflows/buf-binary-size.yaml: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.github/workflows/buf-shadow-sync.yaml: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/github.com/bufbuild/buf@v1.30.1/.github/workflows/buf.yaml: Cannot open: File exists
```
2024-04-12 14:01:16 -07:00
Jeff McCune
4184619afc (#126) Refactor pkg to internal
pkg folder is not needed.  Move everything internal for now.
2024-04-12 13:56:16 -07:00
Jeff McCune
954dbd1ec8 (#126) Refactor id token acquisition to token package
And add a logout command that deletes the token cache.

The token package is intended for subcommands that need to make API
calls to the holos api server, getting a token should be a simple matter
of calling the token.Get() method, which takes minimal dependencies.
2024-04-12 13:15:03 -07:00
Jeff McCune
30b70e76aa (#126) Add login command
This copies the login command from the previous holos cli.  Wire
dependency injection and all the rest of the unnecessary stuff from
kubelogin are removed, streamlined down into a single function that
takes a few oidc related parameters.

This will need to be extracted out into an infrastructure service so
multiple other command line tools can easily re-use it and get the ID
token into the x-oidc-id-token header.
2024-04-12 12:13:33 -07:00
Jeff McCune
ec6d112711 (#126) Remove hydra and kratos databases
No longer needed for dev.
2024-04-12 10:24:26 -07:00
Jeff McCune
e796c6a763 (#126) Default to DATABASE_URL env var 2024-04-12 10:20:13 -07:00
Jeff McCune
be32201294 (#126) Basic User and Organization Ent models
Get rid of the previous UserIdentity model, this is no longer part of
the core domain and instead handled within the context of ZITADEL.
2024-04-12 09:59:40 -07:00
Jeff McCune
5ebc54b5b7 (#124) Go Tools 2024-04-12 09:14:13 -07:00
Jeff McCune
2954a57872 (#120) Fix NATS target namespace
The upstream nats charts don't specify namespaces for each attribute.
This works with helm update, but not helm template which holos uses to
render the yaml.

The missing namespace causes flux to fail.

This patch uses the flux kustomization to add the target namespace to
all resources.
2024-04-10 21:54:58 -07:00
Jeff McCune
df705bd79f (#121) Fix Multiple Charts cause holos render to fail
When rendering a holos component which contains more than one helm chart, rendering fails.  It should succeed.

```
holos render --cluster-name=k2 /home/jeff/workspace/holos-run/holos/docs/examples/platforms/reference/clusters/holos/... --log-level debug
```

```
9:03PM ERR could not execute version=0.64.2 err="could not rename: rename /home/jeff/workspace/holos-run/holos/docs/examples/platforms/reference/clusters/holos/nats/envs/vendor553679311 /home/jeff/workspace/holos-run/holos/docs/examples/platforms/reference/clusters/holos/nats/envs/vendor: file exists" loc=helm.go:145
```

This patch fixes the problem by moving each child item of the temporary
directory charts are installed into.  This avoids the problem of moving
the parent when the parent target already exists.
2024-04-10 21:27:39 -07:00
Jeff McCune
4e8ce3585d (#115) Minor clean up of cue code 2024-04-10 21:21:16 -07:00
Jeff McCune
ab5f17c3d2 (#115) Fix goreleaser
Import modules to take the direct dependency and prevent go mod tidy
from modifying go.mod and go.sum which causes goreleaser to fail.
2024-04-10 19:09:30 -07:00
Jeff McCune
a8918c74d4 (#115) Angular spike - fix make frontend
And install frontend deps.
2024-04-09 21:03:26 -07:00
Jeff McCune
ae5738d82d (#115) Angular with SSR
Executed:

    ng new
    ng add @angular/ssr

Name: holos
Style: CSS
SSR and SSG?: No

ssr added using ng add following https://angular.io/guide/prerendering
2024-04-09 20:52:42 -07:00
Jeff McCune
bb99aedffa (#115) Remove frontend
Clean up for ng new in angular spike.
2024-04-09 20:35:43 -07:00
Jeff McCune
d6ee1864c8 (#116) Tilt for development
Add Tilt back from holos server

Note with this patch the ec-creds.yaml file needs to be applied to the
provisioner and an external secret used to sync the image pull creds.

With this patch the dev instance is accessible behind the auth proxy.
pgAdmin also works from the Tilt UI.

https://jeff.holos.dev.k2.ois.run/app/start
2024-04-09 20:26:37 -07:00
Jeff McCune
8a4be66277 (#113) Fix goreleaser try 4
Please check in your pipeline what can be changing the following files:
  M go.sum
2024-04-09 16:48:21 -07:00
Jeff McCune
79ce2f8458 (#113) Fix goreleaser try 3 2024-04-09 16:35:38 -07:00
Jeff McCune
3d4ae44ddd (#113) Fix goreleaser try 2
goreleaser fails with Failure: plugin connect-query: could not find protoc plugin for name connect-query - please make sure protoc-gen-connect-query is installed and present on your $PATH
2024-04-09 16:23:35 -07:00
Jeff McCune
1efb1faa40 (#113) Fix goreleaser
git executable must come before actions checkout
2024-04-09 16:04:42 -07:00
Jeff McCune
bfd6a56397 (#113) Fix actions workflows 2024-04-09 15:57:31 -07:00
Jeff McCune
a788f6d8e8 (#112) Refactor config flag handling
Remove the server.Config struct, not needed.  Remove the app struct and
move the configuration to the main holos.Config.ServerConfig.

Add flags specific to server configuration.

With this patch logging is simplified.  Subcommands have a handle on the
top level holos.Config and can get a fully configured logger from
cfg.Logger() after flag parsing happens.
2024-04-09 11:42:24 -07:00
Jeff McCune
80fa91d74d (#112) Rename wrapper package to errors
The wrapper package name doesn't indicate what it's for.  Rename to
errors and delegate to the standard library.
2024-04-08 20:53:58 -07:00
Jeff McCune
db34562e9a (#112) Get tests passing 2024-04-08 20:53:57 -07:00
Jeff McCune
d6af089ab3 (#112) Rename package core to app
Disambiguate the term `core` which should mean the core domain.  The app
is a supporting domain concerned with logging and configuration
initialization early in the life cycle.
2024-04-08 20:53:57 -07:00
Jeff McCune
b3a70c5911 (#112) Copy holos-server to holos server subcommand
From holos-server commit da35fe966ded2098fe069293ec30864775a6c4f0

Compiles but needs cleanup
2024-04-08 20:53:25 -07:00
Jeff McCune
bf5765c9cb (#110) Update ZITADEL to v2.49.1 from v2.46.0
Attempt to resolve issue where `/oauth/v2/keys` returns `{"keys": []}`
causing id token verification failures.

Closes: #110
2024-04-07 17:20:10 -07:00
Jeff McCune
6c7697648c (#110) Add runbook to take a full database backup
This runbook documents how to write a full database backup to a blank S3
bucket given an existing postgrescluster resource with a live, running
database.

The pgo controller needs to remove and re-create the repo for the backup
to succeed, otherwise it complains about a missing file expected from a
previous backup.
2024-04-07 17:20:07 -07:00
Jeff McCune
04158485c7 (#96) Do not expire ZITADEL signing public key
The public key needs to be configured along with the signing key.
2024-04-05 10:52:36 -07:00
Jeff McCune
cf83c77280 (#96) Do not expire ZITADEL signing private key
Without this patch users encounter an error from istio because it does
not have a valid Jwks from ZITADEL to verify the request when processing
a `RequestAuthentication` policy.

Fixes error `AuthProxy JWKS Error - Jwks doesn't have key to match kid or alg from Jwt`.

Occurs when accessing a protected URL for the first time after tokens have expired.
2024-04-04 15:56:00 -07:00
Jeff McCune
6e545b13dd (#104) Deploy crunchy monitoring stack for ZITADEL
Not exposed via the ingress gateway, but accessible via

    kubectl port-forward svc/crunchy-grafana 3000

Refer to [day two monitoring][1].  This is pretty much a straight copy
of the upstream kustomize configs at [2].

[1]: https://access.crunchydata.com/documentation/postgres-operator/5.5/tutorials/day-two/monitoring
[2]: https://github.com/CrunchyData/postgres-operator-examples/tree/main/kustomize/monitoring
2024-04-04 15:40:07 -07:00
Jeff McCune
bf258a1f41 (#104) Enable monitoring for ZITADEL postgres
This patch enables the monitoring configuration for the ZITADEL postgres
cluster.

Refer to: https://access.crunchydata.com/documentation/postgres-operator/5.5/tutorials/day-two/monitoring

Integrating with:
https://github.com/CrunchyData/postgres-operator-examples/tree/main/kustomize/monitoring
which will become a separate holos component instance.
2024-04-03 22:26:38 -07:00
Jeff McCune
6f06c73d6f (#85) Initial addition of kube-prometheus-stack
Grafana does not yet have the istio sidecar.  Prometheus is accessible
through the auth proxy.  Cert manager is added to the workload clusters
so tls certs can be issued for webhooks, the kube-prom-stack helm chart
uses cert manager for this purpose.

With this patch Grafana is integrated with OIDC and I'm able to log in
as an Administrator.
2024-04-03 21:29:26 -07:00
Jeff McCune
a689c53a9c (#47) v0.62.1 - Projects v1alpha1 milestone complete 2024-04-03 15:32:34 -07:00
Jeff McCune
58cdda1d35 Merge pull request #100 from holos-run/jeff/47-iam-v2
(#47) Remove the prod-iam-zitadel namespace
2024-04-03 15:23:48 -07:00
Jeff McCune
bcb02b5c5c (#47) Remove the prod-iam-zitadel namespace
No longer needed, cluster has moved to prod-iam namespace.
2024-04-03 15:10:30 -07:00
Jeff McCune
0736c7de1a (#47) Bind ALL VirtualServices to the default gateway
Problem:
The VirtualService that catches auth routes for paths, e.g.
`/holos/authproxy/istio-ingress` is bound to the default gateway which
no longer exists because it has no hosts.

Solution:
It's unnecessary and complicated to create a Gateway for every project.
Instead, put all server entries into one `default` gateway and
consolidate the list using CUE.

Result:
It's easier to reason about this system.  There is only one ingress
gateway, `default` and everything gets added to it.  VirtualServices
need only bind to this gateway, which has a hosts entry appropriately
namespaced for the project.
2024-04-03 14:56:40 -07:00
Jeff McCune
28be9f9fbb (#47) Use the project specific Gateway
The login service is unavailable because the wrong gateway is used.
When using projects the VS needs to attach to the correct Gateway.
2024-04-03 12:59:48 -07:00
Jeff McCune
647681de38 (#99) Restore backups from prod-iam namespace
This patch configures the standby cluster to restore backups from the
prod-iam namespace instead of the prod-iam-zitadel namespace.
2024-04-03 12:30:12 -07:00
Jeff McCune
81beb5c539 (#47) Restore ZITADEL from existing backups
Problem:
The ZITADEL database isn't restoring into the prod-iam namespace after
moving from prod-iam-zitadel because no backup exists at the bucket
path.

Solution:
Hard-code the path to the old namespace to restore the database.  We'll
figure out how to move the backups to the new location in a follow up
change.
2024-04-03 11:44:16 -07:00
Jeff McCune
5c1e0a29c8 (#47) Have Ceph depend on secret stores
Another kustomization reconciling too early.
2024-04-03 11:22:15 -07:00
Jeff McCune
01ac5276a9 (#47) Have Gateway depend on secret stores
The `prod-platform-gateway` kustomization is reconciling early:

ExternalSecret/istio-ingress/argocd.ois.run dry-run failed: failed to
get API group resources: unable to retrieve the complete list of server
APIs: external-secrets.io/v1beta1: the server could not find the
requested resource
2024-04-03 11:20:15 -07:00
Jeff McCune
e40594ad8e (#47) Move ZITADEL to prod-iam project namespace
This patch moves ZITADEL from the prod-iam-zitadel namespace to the
projects managed prod-iam namespace, which is the prod environment of
the prod stage of the iam project.
2024-04-03 11:06:55 -07:00
Jeff McCune
bc9c6a622a (#97) Increase ZITADEL pgdata volume to 20Gi
Problem:

```
❯ k exec zitadel-pgha1-4npq-0 -it -- bash
Defaulted container "database" out of: database, replication-cert-copy, pgbackrest, pgbackrest-config, postgres-startup (init), nss-wrapper-init (init)
bash-4.4$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         119G   51G   68G  43% /
tmpfs            64M     0   64M   0% /dev
/dev/rbd3       9.8G  9.8G     0 100% /pgdata
/dev/sda6       119G   51G   68G  43% /tmp
tmpfs            16G   24K   16G   1% /pgconf/tls
tmpfs            16G   24K   16G   1% /etc/database-containerinfo
tmpfs            16G   16K   16G   1% /etc/patroni
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G   28K   16G   1% /etc/pgbackrest/conf.d
tmpfs            16G   12K   16G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           7.9G     0  7.9G   0% /proc/acpi
tmpfs           7.9G     0  7.9G   0% /proc/scsi
tmpfs           7.9G     0  7.9G   0% /sys/firmware
```
2024-04-03 10:09:49 -07:00
Jeff McCune
17f22199b7 (#86) ArgoCD - Disable Dex
Not needed
2024-04-02 15:47:22 -07:00
Jeff McCune
7e93fe4535 (#86) ArgoCD
Using the Helm chart so we can inject the istio sidecar with a kustomize
patch and tweak the configs for OIDC integration.

Login works, istio sidecar is injected.  ArgoCD can only be configured
with one domain unfortunately, it's not accessible at argocd.ois.run,
only argocd.k2.ois.run (or whatever cluster it's installed into).

Ideally it would use the Host header but it does not.

RBAC is not implemented but the User Info endpoint does have group
membership so this shouldn't be a problem to implement.
2024-04-02 15:33:47 -07:00
Jeff McCune
2e98df3572 (#86) ArgoCD in prod-platform project namespace
Deploys using the official release yaml.
2024-04-02 13:34:03 -07:00
Jeff McCune
3b561de413 (#93) Custom AuthPolicy rules for vault
This patch defines a #AuthPolicyRules struct which excludes hosts from
the blanket auth policy and includes them in specialized auth policies.
The purpose is to handle special cases like vault requests which have an
`X-Vault-Token` and `X-Vault-Request` header.

Vault does not use jwts so we cannot verify them in the mesh, have to
pass them along to the backend.

Closes: #93
2024-04-02 12:54:31 -07:00
Jeff McCune
0d0dae8742 (#89) Disable project auth proxies by default
Focus on the ingress gateway auth proxy for now and see how far it gets
us.
2024-04-01 21:48:08 -07:00
Jeff McCune
61b4b5bd17 (#89) Refactor auth proxy callbacks
The ingress gateway auth proxy callback conflicts with the project stage
auth proxy callback for the same backend Host: header value.

This patch disambiguates them by the namespace the auth proxy resides
in.
2024-04-01 21:37:52 -07:00
1480 changed files with 293404 additions and 53620 deletions

107
.cspell.json Normal file
View File

@@ -0,0 +1,107 @@
{
"version": "0.2",
"language": "en",
"enableFiletypes": [
"mdx"
],
"words": [
"applicationset",
"argoproj",
"authcode",
"authpolicy",
"authproxy",
"authroutes",
"buildplan",
"cainjector",
"clsx",
"clusterissuer",
"cookiesecret",
"coredns",
"CRD's",
"crds",
"creds",
"crossplane",
"cuecontext",
"cuelang",
"devicecode",
"dnsmasq",
"dscacheutil",
"entgo",
"errgroup",
"fieldmaskpb",
"flushcache",
"gendoc",
"ghaction",
"gitops",
"godoc",
"golangci",
"goreleaser",
"grpcreflect",
"grpcurl",
"holos",
"holoslogger",
"httpbin",
"Infima",
"isatty",
"istiod",
"jetstack",
"Jsonnet",
"killall",
"kubeadm",
"kubeconfig",
"kubelogin",
"Kustomization",
"Kustomizations",
"kustomize",
"ldflags",
"libnss",
"loadbalancer",
"mattn",
"mktemp",
"Multicluster",
"mxcl",
"myhostname",
"nameserver",
"organizationconnect",
"orgid",
"otelconnect",
"Parentspanid",
"pflag",
"PKCE",
"platformconnect",
"podinfo",
"promhttp",
"protobuf",
"protojson",
"Pulumi",
"putenv",
"quickstart",
"retryable",
"ropc",
"SECRETKEY",
"secretstores",
"spanid",
"spiffe",
"startupapicheck",
"structpb",
"systemconnect",
"tablewriter",
"Tiltfile",
"timestamppb",
"Timoni",
"tlsclientconfig",
"tokencache",
"Tokener",
"Traceid",
"traefik",
"uibutton",
"untar",
"Upsert",
"urandom",
"usecases",
"userconnect",
"userdata",
"zerolog",
"zitadel"
]
}

View File

@@ -18,13 +18,18 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 20
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: stable
- name: Provide unzip for Helm
run: sudo apt update && sudo apt -qq -y install curl zip unzip tar bzip2
- name: Install Packages
run: sudo apt update && sudo apt -qq -y install git curl zip unzip tar bzip2 make
- name: Set up Helm
uses: azure/setup-helm@v4
@@ -32,5 +37,10 @@ jobs:
- name: Set up Kubectl
uses: azure/setup-kubectl@v3
- name: Install Tools
run: |
set -x
make tools
- name: Test
run: ./scripts/test

57
.github/workflows/dev-deploy.yaml vendored Normal file
View File

@@ -0,0 +1,57 @@
name: Dev Deploy
on:
push:
branches: ['main', 'dev-deploy']
jobs:
deploy:
name: Deploy
runs-on: ubuntu-latest
steps:
## Not needed on ubuntu-latest
# - name: Provide GPG and Git
# run: sudo apt update && sudo apt -qq -y install gnupg git curl zip unzip tar bzip2 make jq
## Not needed on ubuntu-latest
# - name: Provide Holos Dependencies
# run: |
# sudo mkdir -p -m 755 /etc/apt/keyrings
# curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
# sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg
# echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
# sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list
# sudo apt update
# sudo apt install -qq -y kubectl
# curl -fsSL -o- https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
# Must come after git executable is provided
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-go@v5
with:
go-version: '1.22.x'
- uses: ko-build/setup-ko@v0.7
env:
KO_DOCKER_REPO: quay.io/holos-run/holos
- name: Setup SSH
run: |
mkdir -p ~/.ssh
echo "${{ secrets.DEPLOY_SSH_PRIVATE_KEY }}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh-keyscan github.com >> ~/.ssh/known_hosts
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
- name: make dev-deploy
env:
auth_user: holos-run+pusher
auth_token: ${{ secrets.QUAY_TOKEN }}
run: |
echo "${auth_token}" | ko login quay.io --username "${auth_user}" --password-stdin
make dev-deploy

30
.github/workflows/golangci-lint.yaml vendored Normal file
View File

@@ -0,0 +1,30 @@
name: golangci-lint
on:
push:
branches:
- main
- test
pull_request:
types: [opened, synchronize]
permissions:
# Required: allow read access to the content for analysis.
contents: read
# Optional: allow read access to pull request. Use with `only-new-issues` option.
pull-requests: read
# Optional: allow write access to checks to allow the action to annotate code in the PR.
checks: write
jobs:
golangci:
name: lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: stable
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
with:
version: v1.60

View File

@@ -13,16 +13,30 @@ permissions:
contents: read
jobs:
golangci:
lint:
name: lint
runs-on: gha-rs
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 20
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: stable
cache: false
- name: golangci-lint
uses: golangci/golangci-lint-action@v4
with:
version: latest
## Not needed on ubuntu-latest
# - name: Install Packages
# run: sudo apt update && sudo apt -qq -y install git curl zip unzip tar bzip2 make
- name: Install Tools
run: make tools
- name: Lint
# golangci-lint runs in a separate workflow.
run: make lint -o golangci-lint

View File

@@ -12,25 +12,48 @@ permissions:
jobs:
goreleaser:
runs-on: gha-rs
runs-on: ubuntu-latest
steps:
- name: Provide GPG and Git
run: sudo apt update && sudo apt -qq -y install gnupg git
## Not needed on ubuntu-latest
# Must come before Checkout, otherwise goreleaser fails
# - name: Provide GPG and Git
# run: sudo apt update && sudo apt -qq -y install gnupg git curl zip unzip tar bzip2 make
# Must come after git executable is provided
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 20
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: stable
# Necessary to run these outside of goreleaser, otherwise
# /home/runner/_work/holos/holos/internal/frontend/node_modules/.bin/protoc-gen-connect-query is not in PATH
- name: Install Tools
run: |
set -x
make tools
- name: Import GPG key
uses: crazy-max/ghaction-import-gpg@v6
with:
gpg_private_key: ${{ secrets.GPG_CODE_SIGNING_SECRETKEY }}
passphrase: ${{ secrets.GPG_CODE_SIGNING_PASSPHRASE }}
- name: List keys
run: gpg -K
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: stable
- name: Git diff
run: git diff
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v5
with:

12
.gitignore vendored
View File

@@ -1,8 +1,14 @@
bin/
vendor/
/bin/
.idea/
coverage.out
dist/
/dist/
*.hold/
/deploy/
.vscode/
tmp/
.DS_*
# In case we run through the tutorial in this directory.
/holos-k3d/
/holos-infra/
node_modules/

View File

@@ -10,10 +10,8 @@ version: 1
before:
hooks:
# You may remove this if you don't use go modules.
- go mod tidy
# you may remove this if you don't need go generate
- go generate ./...
- go mod tidy
builds:
- main: ./cmd/holos
@@ -23,6 +21,9 @@ builds:
- linux
- windows
- darwin
goarch:
- amd64
- arm64
signs:
- artifacts: checksum

13
.ko.yaml Normal file
View File

@@ -0,0 +1,13 @@
# Refer to https://ko.build/configuration/#overriding-go-build-settings
builds:
- id: holos
dir: .
main: ./cmd/holos
env:
- GOPRIVATE=github.com/holos-run/\*
ldflags:
- -s
- -w
- -X
# Makefile provides GIT_DETAIL and GIT_SUFFIX.
- github.com/holos-run/holos/version.GitDescribe={{.Env.GIT_DETAIL}}{{.Env.GIT_SUFFIX}}

8
Dockerfile Normal file
View File

@@ -0,0 +1,8 @@
FROM quay.io/holos-run/debian:bullseye AS final
USER root
WORKDIR /app
ADD bin bin
RUN chown -R app: /app
# Kubernetes requires the user to be numeric
USER 8192
ENTRYPOINT bin/holos server

202
LICENSE Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

View File

@@ -4,19 +4,24 @@ PROJ=holos
ORG_PATH=github.com/holos-run
REPO_PATH=$(ORG_PATH)/$(PROJ)
VERSION := $(shell cat pkg/version/embedded/major pkg/version/embedded/minor pkg/version/embedded/patch | xargs printf "%s.%s.%s")
VERSION := $(shell cat version/embedded/major version/embedded/minor version/embedded/patch | xargs printf "%s.%s.%s")
BIN_NAME := holos
DOCKER_REPO=quay.io/openinfrastructure/holos
DOCKER_REPO=quay.io/holos-run/holos
IMAGE_NAME=$(DOCKER_REPO)
$( shell mkdir -p bin)
# For buf plugin protoc-gen-connect-es
export PATH := $(PWD)/internal/frontend/holos/node_modules/.bin:$(PATH)
GIT_COMMIT=$(shell git rev-parse HEAD)
GIT_SUFFIX=$(shell test -n "`git status --porcelain`" && echo "-dirty" || echo "")
GIT_DETAIL=$(shell git describe --tags HEAD)
GIT_TREE_STATE=$(shell test -n "`git status --porcelain`" && echo "dirty" || echo "clean")
BUILD_DATE=$(shell date -Iseconds)
LD_FLAGS="-w -X ${ORG_PATH}/${PROJ}/pkg/version.GitCommit=${GIT_COMMIT} -X ${ORG_PATH}/${PROJ}/pkg/version.GitTreeState=${GIT_TREE_STATE} -X ${ORG_PATH}/${PROJ}/pkg/version.BuildDate=${BUILD_DATE}"
LD_FLAGS="-w -X ${ORG_PATH}/${PROJ}/version.GitDescribe=${GIT_DETAIL}${GIT_SUFFIX} -X ${ORG_PATH}/${PROJ}/version.GitCommit=${GIT_COMMIT} -X ${ORG_PATH}/${PROJ}/version.GitTreeState=${GIT_TREE_STATE} -X ${ORG_PATH}/${PROJ}/version.BuildDate=${BUILD_DATE}"
.PHONY: default
default: test
@@ -43,33 +48,38 @@ bumpmajor: ## Bump the major version.
show-version: ## Print the full version.
@echo $(VERSION)
.PHONY: tag
tag: ## Tag a release
git tag v$(VERSION)
.PHONY: tidy
tidy: ## Tidy go module.
go mod tidy
.PHONY: fmt
fmt: ## Format code.
cd docs/examples && cue fmt ./...
cd internal/generate/platforms && cue fmt ./...
go fmt ./...
.PHONY: vet
vet: ## Vet Go code.
go vet ./...
.PHONY: gencue
gencue: ## Generate CUE definitions
cd docs/examples && cue get go github.com/holos-run/holos/api/...
.PHONY: generate
generate: ## Generate code.
go generate ./...
.PHONY: build
build: generate ## Build holos executable.
build: ## Build holos executable.
@echo "building ${BIN_NAME} ${VERSION}"
@echo "GOPATH=${GOPATH}"
go build -trimpath -o bin/$(BIN_NAME) -ldflags $(LD_FLAGS) $(REPO_PATH)/cmd/$(BIN_NAME)
linux: ## Build holos executable for tilt.
@echo "building ${BIN_NAME}.linux ${VERSION}"
@echo "GOPATH=${GOPATH}"
GOOS=linux go build -trimpath -o bin/$(BIN_NAME).linux -ldflags $(LD_FLAGS) $(REPO_PATH)/cmd/$(BIN_NAME)
.PHONY: install
install: build ## Install holos to GOPATH/bin
install bin/$(BIN_NAME) $(shell go env GOPATH)/bin/$(BIN_NAME)
@@ -82,10 +92,16 @@ clean: ## Clean executables.
test: ## Run tests.
scripts/test
.PHONY: lint
lint: ## Run linters.
.PHONY: golangci-lint
golangci-lint:
golangci-lint run
.PHONY: lint
lint: golangci-lint ## Run linters.
buf lint
cd internal/frontend/holos && ng lint
./hack/cspell
.PHONY: coverage
coverage: test ## Test coverage profile.
go tool cover -html=coverage.out
@@ -94,6 +110,46 @@ coverage: test ## Test coverage profile.
snapshot: ## Go release snapshot
goreleaser release --snapshot --clean
.PHONY: tools
tools: go-deps frontend-deps website-deps ## install tool dependencies
.PHONY: go-deps
go-deps: ## tool versions pinned in tools.go
go install cuelang.org/go/cmd/cue
go install github.com/bufbuild/buf/cmd/buf
go install github.com/fullstorydev/grpcurl/cmd/grpcurl
go install google.golang.org/protobuf/cmd/protoc-gen-go
go install connectrpc.com/connect/cmd/protoc-gen-connect-go
go install honnef.co/go/tools/cmd/staticcheck
go install golang.org/x/tools/cmd/godoc
go install github.com/princjef/gomarkdoc/cmd/gomarkdoc
go install github.com/google/ko
# curl https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | bash
.PHONY: frontend-deps
frontend-deps: ## Install Angular deps for go generate
cd internal/frontend/holos && npm install
.PHONY: website-deps
website-deps: ## Install Docusaurus deps for go generate
cd doc/website && npm install
.PHONY: image # refer to .ko.yaml as well
image: ## Container image build for workflows/publish.yaml
KO_DOCKER_REPO=$(DOCKER_REPO) GIT_DETAIL=$(GIT_DETAIL) GIT_SUFFIX=$(GIT_SUFFIX) ko build --platform=all --bare ./cmd/holos --tags $(GIT_DETAIL)$(GIT_SUFFIX) --tags latest
.PHONY: prod-deploy
prod-deploy: install image ## deploy to PROD
GIT_DETAIL=$(GIT_DETAIL) GIT_SUFFIX=$(GIT_SUFFIX) bash ./hack/deploy
.PHONY: dev-deploy
dev-deploy: install image ## deploy to dev
GIT_DETAIL=$(GIT_DETAIL) GIT_SUFFIX=$(GIT_SUFFIX) bash ./hack/deploy-dev
.PHONY: website
website: ## Build website
./hack/build-website
.PHONY: help
help: ## Display this help menu.
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-20s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)

35
README.md Normal file
View File

@@ -0,0 +1,35 @@
## Holos - A Holistic Development Platform
<img width="50%"
align="right"
style="display: block; margin: 40px auto;"
src="https://openinfrastructure.co/blog/2016/02/27/logo/logorectangle.png">
Building and maintaining a software development platform is a complex and time
consuming endeavour. Organizations often dedicate a team of 3-4 who need 6-12
months to build the platform.
Holos is a tool and a reference platform to reduce the complexity and speed up
the process of building a modern, cloud native software development platform.
- **Accelerate new projects** - Reduce time to market and operational complexity by starting your new project on top of the Holos reference platform.
- **Modernize existing projects** - Incrementally incorporate your existing platform services into Holos for simpler integration.
- **Unified configuration model** - Increase safety and reduce the risk of config changes with CUE.
- **First class Helm and Kustomize support** - Leverage and reuse your existing investment in existing configuration tools such as Helm and Kustomize.
- **Modern Authentication and Authorization** - Holos seamlessly integrates platform identity and access management with zero-trust beyond corp style authorization policy.
## Quick Installation
```console
go install github.com/holos-run/holos/cmd/holos@latest
```
## Docs and Support
The documentation for developing and using Holos is available at: https://holos.run
For discussion and support, [open a discussion](https://github.com/orgs/holos-run/discussions/new/choose).
## License
Holos is licensed under Apache 2.0 as found in the [LICENSE file](LICENSE).

110
Tiltfile Normal file
View File

@@ -0,0 +1,110 @@
# -*- mode: Python -*-
# This Tiltfile manages a Go project with live reload in Kubernetes
listen_port = 3000
metrics_port = 9090
# Use our wrapper to set the kube namespace
if os.getenv('TILT_WRAPPER') != '1':
fail("could not run, ./hack/tilt/bin/tilt was not used to start tilt")
# Resource ids
holos_backend = 'Holos Server'
compile_id = 'Go Build'
# Default Registry.
# See: https://github.com/tilt-dev/tilt.build/blob/master/docs/choosing_clusters.md#manual-configuration
# Note, Tilt will append the image name to the registry uri path
# default_registry('{account}.dkr.ecr.{region}.amazonaws.com/holos-run/holos'.format(account=aws_account, region=aws_region))
# Set a name prefix specific to the user. Multiple developers share the tilt-holos namespace.
developer = os.getenv('USER')
holos_server = 'holos'
# We always develop against the k3d-workload cluster
os.putenv('KUBECONFIG', os.path.abspath('./hack/tilt/kubeconfig'))
# Extensions are open-source, pre-packaged functions that extend Tilt
#
# More info: https://github.com/tilt-dev/tilt-extensions
# More info: https://docs.tilt.dev/extensions.html
load('ext://restart_process', 'docker_build_with_restart')
load('ext://k8s_attach', 'k8s_attach')
load('ext://git_resource', 'git_checkout')
load('ext://uibutton', 'cmd_button')
# Paths edited by the developer Tilt watches to trigger compilation.
# Generated files should be excluded to avoid an infinite build loop.
developer_paths = [
'./cmd',
'./internal/server',
'./internal/ent/schema',
'./frontend/package-lock.json',
'./frontend/src',
'./go.mod',
'./pkg',
'./service/holos',
]
# Builds the holos executable GOOS=linux
local_resource(compile_id, 'make linux', deps=developer_paths)
# Build Docker image
# Tilt will automatically associate image builds with the resource(s)
# that reference them (e.g. via Kubernetes or Docker Compose YAML).
#
# More info: https://docs.tilt.dev/api.html#api.docker_build
#
docker_build_with_restart(
'k3d-registry.holos.localhost:5100/holos',
context='.',
entrypoint=[
'/app/bin/holos.linux',
'server',
'--log-format=text',
'--oidc-issuer=https://login.holos.run',
'--oidc-audience=275804490387516853@holos_quickstart', # auth proxy
'--oidc-audience=270319630705329162@holos_platform', # holos cli
],
dockerfile='./Dockerfile',
only=['./bin'],
# (Recommended) Updating a running container in-place
# https://docs.tilt.dev/live_update_reference.html
live_update=[
# Sync files from host to container
sync('./bin/', '/app/bin/'),
],
)
# Troubleshooting
def resource_name(id):
print('resource: {}'.format(id))
return id.name
workload_to_resource_function(resource_name)
# Customize a Kubernetes resource
# By default, Kubernetes resource names are automatically assigned
# based on objects in the YAML manifests, e.g. Deployment name.
#
# Tilt strives for sane defaults, so calling k8s_resource is
# optional, and you only need to pass the arguments you want to
# override.
#
# More info: https://docs.tilt.dev/api.html#api.k8s_resource
#
k8s_yaml(blob(str(read_file('./hack/tilt/k8s/dev-holos-app/deployment.yaml'))))
# Backend server process
k8s_resource(
workload=holos_server,
new_name=holos_backend,
objects=[],
resource_deps=[compile_id],
links=[
link('https://app.holos.localhost/ui/'.format(developer), "Holos Web UI")
],
)
# Database
print("✨ Tiltfile evaluated")

View File

@@ -0,0 +1,44 @@
package v1alpha2
import "google.golang.org/protobuf/types/known/structpb"
// Label is an arbitrary unique identifier internal to holos itself. The holos
// cli is expected to never write a Label value to rendered output files,
// therefore use a [Label] then the identifier must be unique and internal.
// Defined as a type for clarity and type checking.
//
// A Label is useful to convert a CUE struct to a list, for example producing a list of [APIObject] resources from an [APIObjectMap]. A CUE struct using
// Label keys is guaranteed to not lose data when rendering output because a
// Label is expected to never be written to the final output.
type Label string
// Kind is a kubernetes api object kind. Defined as a type for clarity and type checking.
type Kind string
// APIObject represents the most basic generic form of a single kubernetes api
// object. Represented as a JSON object internally for compatibility between
// tools, for example loading from CUE.
type APIObject structpb.Struct
// APIObjectMap represents the marshalled yaml representation of kubernetes api
// objects. Do not produce an APIObjectMap directly, instead use [APIObjects]
// to produce the marshalled yaml representation from CUE data, then provide the
// result to [HolosComponent].
type APIObjectMap map[Kind]map[Label]string
// APIObjects represents Kubernetes API objects defined directly from CUE code.
// Useful to mix in resources to any kind of [HolosComponent], for example
// adding an ExternalSecret resource to a [HelmChart].
//
// [Kind] must be the resource kind, e.g. Deployment or Service.
//
// [Label] is an arbitrary internal identifier to uniquely identify the resource
// within the context of a `holos` command. Holos will never write the
// intermediate label to rendered output.
//
// Refer to [HolosComponent] which accepts an [APIObjectMap] field provided by
// [APIObjects].
type APIObjects struct {
APIObjects map[Kind]map[Label]APIObject `json:"apiObjects"`
APIObjectMap APIObjectMap `json:"apiObjectMap"`
}

View File

@@ -0,0 +1,96 @@
package v1alpha2
// FilePath represents a file path.
type FilePath string
// FileContent represents file contents.
type FileContent string
// FileContentMap represents a mapping of file paths to file contents. Paths
// are relative to the `holos` output "deploy" directory, and may contain
// sub-directories.
type FileContentMap map[FilePath]FileContent
// BuildPlan represents a build plan for the holos cli to execute. The purpose
// of a BuildPlan is to define one or more [HolosComponent] kinds. For example a
// [HelmChart], [KustomizeBuild], or [KubernetesObjects].
//
// A BuildPlan usually has an additional empty [KubernetesObjects] for the
// purpose of using the [HolosComponent] DeployFiles field to deploy an ArgoCD
// or Flux gitops resource for the holos component.
type BuildPlan struct {
Kind string `json:"kind" cue:"\"BuildPlan\""`
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha2\""`
Spec BuildPlanSpec `json:"spec"`
}
// BuildPlanSpec represents the specification of the build plan.
type BuildPlanSpec struct {
// Disabled causes the holos cli to take no action over the [BuildPlan].
Disabled bool `json:"disabled,omitempty"`
// Components represents multiple [HolosComponent] kinds to manage.
Components BuildPlanComponents `json:"components,omitempty"`
}
type BuildPlanComponents struct {
Resources map[Label]KubernetesObjects `json:"resources,omitempty"`
KubernetesObjectsList []KubernetesObjects `json:"kubernetesObjectsList,omitempty"`
HelmChartList []HelmChart `json:"helmChartList,omitempty"`
KustomizeBuildList []KustomizeBuild `json:"kustomizeBuildList,omitempty"`
}
// HolosComponent defines the fields common to all holos component kinds. Every
// holos component kind should embed HolosComponent.
type HolosComponent struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind"`
// APIVersion represents the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha2\""`
// Metadata represents data about the holos component such as the Name.
Metadata Metadata `json:"metadata"`
// APIObjectMap holds the marshalled representation of api objects. Useful to
// mix in resources to each HolosComponent type, for example adding an
// ExternalSecret to a HelmChart HolosComponent. Refer to [APIObjects].
APIObjectMap APIObjectMap `json:"apiObjectMap,omitempty"`
// DeployFiles represents file paths relative to the cluster deploy directory
// with the value representing the file content. Intended for defining the
// ArgoCD Application resource or Flux Kustomization resource from within CUE,
// but may be used to render any file related to the build plan from CUE.
DeployFiles FileContentMap `json:"deployFiles,omitempty"`
// Kustomize represents a kubectl kustomize build post-processing step.
Kustomize `json:"kustomize,omitempty"`
// Skip causes holos to take no action regarding this component.
Skip bool `json:"skip" cue:"bool | *false"`
}
// Metadata represents data about the holos component such as the Name.
type Metadata struct {
// Name represents the name of the holos component.
Name string `json:"name"`
// Namespace is the primary namespace of the holos component. A holos
// component may manage resources in multiple namespaces, in this case
// consider setting the component namespace to default.
//
// This field is optional because not all resources require a namespace,
// particularly CRD's and DeployFiles functionality.
// +optional
Namespace string `json:"namespace,omitempty"`
}
// Kustomize represents resources necessary to execute a kustomize build.
// Intended for at least two use cases:
//
// 1. Process a [KustomizeBuild] [HolosComponent] which represents raw yaml
// file resources in a holos component directory.
// 2. Post process a [HelmChart] [HolosComponent] to inject istio, patch jobs,
// add custom labels, etc...
type Kustomize struct {
// KustomizeFiles holds file contents for kustomize, e.g. patch files.
KustomizeFiles FileContentMap `json:"kustomizeFiles,omitempty"`
// ResourcesFile is the file name used for api objects in kustomization.yaml
ResourcesFile string `json:"resourcesFile,omitempty"`
}

View File

@@ -0,0 +1,11 @@
package v1alpha2
const (
APIVersion = "v1alpha2"
BuildPlanKind = "BuildPlan"
HelmChartKind = "HelmChart"
// ChartDir is the directory name created in the holos component directory to cache a chart.
ChartDir = "vendor"
// ResourcesFile is the file name used to store component output when post-processing with kustomize.
ResourcesFile = "resources.yaml"
)

44
api/core/v1alpha2/core.go Normal file
View File

@@ -0,0 +1,44 @@
package v1alpha2
import "google.golang.org/protobuf/types/known/structpb"
type PlatformMetadata struct {
// Name represents the Platform name.
Name string `json:"name"`
}
// Platform represents a platform to manage. A Platform resource informs holos
// which components to build. The platform resource also acts as a container
// for the platform model form values provided by the PlatformService. The
// primary use case is to collect the cluster names, cluster types, platform
// model, and holos components to build into one resource.
type Platform struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind" cue:"\"Platform\""`
// APIVersion represents the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha2\""`
// Metadata represents data about the object such as the Name.
Metadata PlatformMetadata `json:"metadata"`
// Spec represents the specification.
Spec PlatformSpec `json:"spec"`
}
// PlatformSpec represents the specification of a Platform. Think of a platform
// specification as a list of platform components to apply to a list of
// kubernetes clusters combined with the user-specified Platform Model.
type PlatformSpec struct {
// Model represents the platform model holos gets from from the
// PlatformService.GetPlatform rpc method and provides to CUE using a tag.
Model structpb.Struct `json:"model"`
// Components represents a list of holos components to manage.
Components []PlatformSpecComponent `json:"components"`
}
// PlatformSpecComponent represents a holos component to build or render.
type PlatformSpecComponent struct {
// Path is the path of the component relative to the platform root.
Path string `json:"path"`
// Cluster is the cluster name to provide when rendering the component.
Cluster string `json:"cluster"`
}

26
api/core/v1alpha2/doc.go Normal file
View File

@@ -0,0 +1,26 @@
// Package v1alpha2 contains the core API contract between the holos cli and CUE
// configuration code. Platform designers, operators, and software developers
// use this API to write configuration in CUE which `holos` loads. The overall
// shape of the API defines imperative actions `holos` should carry out to
// render the complete yaml that represents a Platform.
//
// [Platform] defines the complete configuration of a platform. With the holos
// reference platform this takes the shape of one management cluster and at
// least two workload cluster. Each cluster has multiple [HolosComponent]
// resources applied to it.
//
// Each holos component path, e.g. `components/namespaces` produces exactly one
// [BuildPlan] which in turn contains a set of [HolosComponent] kinds.
//
// The primary kinds of [HolosComponent] are:
//
// 1. [HelmChart] to render config from a helm chart.
// 2. [KustomizeBuild] to render config from [Kustomize]
// 3. [KubernetesObjects] to render [APIObjects] defined directly in CUE
// configuration.
//
// Note that Holos operates as a data pipeline, so the output of a [HelmChart]
// may be provided to [Kustomize] for post-processing.
package v1alpha2
//go:generate ../../../hack/gendoc

38
api/core/v1alpha2/helm.go Normal file
View File

@@ -0,0 +1,38 @@
package v1alpha2
// HelmChart represents a holos component which wraps around an upstream helm
// chart. Holos orchestrates helm by providing values obtained from CUE,
// renders the output using `helm template`, then post-processes the helm output
// yaml using the general functionality provided by [HolosComponent], for
// example [Kustomize] post-rendering and mixing in additional kubernetes api
// objects.
type HelmChart struct {
HolosComponent `json:",inline"`
Kind string `json:"kind" cue:"\"HelmChart\""`
// Chart represents a helm chart to manage.
Chart Chart `json:"chart"`
// ValuesContent represents the values.yaml file holos passes to the `helm
// template` command.
ValuesContent string `json:"valuesContent"`
// EnableHooks enables helm hooks when executing the `helm template` command.
EnableHooks bool `json:"enableHooks" cue:"bool | *false"`
}
// Chart represents a helm chart.
type Chart struct {
// Name represents the chart name.
Name string `json:"name"`
// Version represents the chart version.
Version string `json:"version"`
// Release represents the chart release when executing helm template.
Release string `json:"release"`
// Repository represents the repository to fetch the chart from.
Repository Repository `json:"repository,omitempty"`
}
// Repository represents a helm chart repository.
type Repository struct {
Name string `json:"name"`
URL string `json:"url"`
}

View File

@@ -0,0 +1,10 @@
package v1alpha2
const KubernetesObjectsKind = "KubernetesObjects"
// KubernetesObjects represents a [HolosComponent] composed of Kubernetes API
// objects provided directly from CUE using [APIObjects].
type KubernetesObjects struct {
HolosComponent `json:",inline"`
Kind string `json:"kind" cue:"\"KubernetesObjects\""`
}

View File

@@ -0,0 +1,8 @@
package v1alpha2
// KustomizeBuild represents a [HolosComponent] that renders plain yaml files in
// the holos component directory using `kubectl kustomize build`.
type KustomizeBuild struct {
HolosComponent `json:",inline"`
Kind string `json:"kind" cue:"\"KustomizeBuild\""`
}

View File

@@ -0,0 +1,47 @@
package v1alpha3
import "google.golang.org/protobuf/types/known/structpb"
// InternalLabel is an arbitrary unique identifier internal to holos itself.
// The holos cli is expected to never write a InternalLabel value to rendered
// output files, therefore use a [InternalLabel] when the identifier must be
// unique and internal. Defined as a type for clarity and type checking.
//
// A InternalLabel is useful to convert a CUE struct to a list, for example
// producing a list of [APIObject] resources from an [APIObjectMap]. A CUE
// struct using InternalLabel keys is guaranteed to not lose data when rendering
// output because a InternalLabel is expected to never be written to the final
// output.
type InternalLabel string
// Kind is a kubernetes api object kind. Defined as a type for clarity and type
// checking.
type Kind string
// APIObject represents the most basic generic form of a single kubernetes api
// object. Represented as a JSON object internally for compatibility between
// tools, for example loading from CUE.
type APIObject structpb.Struct
// APIObjectMap represents the marshalled yaml representation of kubernetes api
// objects. Do not produce an APIObjectMap directly, instead use [APIObjects]
// to produce the marshalled yaml representation from CUE data, then provide the
// result to [Component].
type APIObjectMap map[Kind]map[InternalLabel]string
// APIObjects represents Kubernetes API objects defined directly from CUE code.
// Useful to mix in resources to any kind of [Component], for example
// adding an ExternalSecret resource to a [HelmChart].
//
// [Kind] must be the resource kind, e.g. Deployment or Service.
//
// [InternalLabel] is an arbitrary internal identifier to uniquely identify the resource
// within the context of a `holos` command. Holos will never write the
// intermediate label to rendered output.
//
// Refer to [Component] which accepts an [APIObjectMap] field provided by
// [APIObjects].
type APIObjects struct {
APIObjects map[Kind]map[InternalLabel]APIObject `json:"apiObjects"`
APIObjectMap APIObjectMap `json:"apiObjectMap"`
}

View File

@@ -0,0 +1,52 @@
package v1alpha3
// FilePath represents a file path.
type FilePath string
// FileContent represents file contents.
type FileContent string
// FileContentMap represents a mapping of file paths to file contents.
type FileContentMap map[FilePath]FileContent
// BuildPlan represents a build plan for the holos cli to execute. The purpose
// of a BuildPlan is to define one or more [Component] kinds. For example a
// [HelmChart], [KustomizeBuild], or [KubernetesObjects].
//
// A BuildPlan usually has an additional empty [KubernetesObjects] for the
// purpose of using the [Component] DeployFiles field to deploy an ArgoCD
// or Flux gitops resource for the holos component.
type BuildPlan struct {
Kind string `json:"kind" cue:"\"BuildPlan\""`
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha3\""`
Spec BuildPlanSpec `json:"spec"`
}
// BuildPlanSpec represents the specification of the build plan.
type BuildPlanSpec struct {
// Disabled causes the holos cli to take no action over the [BuildPlan].
Disabled bool `json:"disabled,omitempty"`
// Components represents multiple [HolosComponent] kinds to manage.
Components BuildPlanComponents `json:"components,omitempty"`
}
type BuildPlanComponents struct {
Resources map[InternalLabel]KubernetesObjects `json:"resources,omitempty"`
KubernetesObjectsList []KubernetesObjects `json:"kubernetesObjectsList,omitempty"`
HelmChartList []HelmChart `json:"helmChartList,omitempty"`
KustomizeBuildList []KustomizeBuild `json:"kustomizeBuildList,omitempty"`
}
// Kustomize represents resources necessary to execute a kustomize build.
// Intended for at least two use cases:
//
// 1. Process a [KustomizeBuild] [Component] which represents raw yaml
// file resources in a holos component directory.
// 2. Post process a [HelmChart] [Component] to inject istio, patch jobs,
// add custom labels, etc...
type Kustomize struct {
// KustomizeFiles holds file contents for kustomize, e.g. patch files.
KustomizeFiles FileContentMap `json:"kustomizeFiles,omitempty"`
// ResourcesFile is the file name used for api objects in kustomization.yaml
ResourcesFile string `json:"resourcesFile,omitempty"`
}

View File

@@ -0,0 +1,43 @@
package v1alpha3
// Component defines the fields common to all holos component kinds. Every
// holos component kind should embed Component.
type Component struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind"`
// APIVersion represents the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"\"v1alpha3\""`
// Metadata represents data about the holos component such as the Name.
Metadata Metadata `json:"metadata"`
// APIObjectMap holds the marshalled representation of api objects. Useful to
// mix in resources to each Component type, for example adding an
// ExternalSecret to a [HelmChart] Component. Refer to [APIObjects].
APIObjectMap APIObjectMap `json:"apiObjectMap,omitempty"`
// DeployFiles represents file paths relative to the cluster deploy directory
// with the value representing the file content. Intended for defining the
// ArgoCD Application resource or Flux Kustomization resource from within CUE,
// but may be used to render any file related to the build plan from CUE.
DeployFiles FileContentMap `json:"deployFiles,omitempty"`
// Kustomize represents a kubectl kustomize build post-processing step.
Kustomize `json:"kustomize,omitempty"`
// Skip causes holos to take no action regarding this component.
Skip bool `json:"skip" cue:"bool | *false"`
}
// Metadata represents data about the object such as the Name.
type Metadata struct {
// Name represents the name of the holos component.
Name string `json:"name"`
// Namespace is the primary namespace of the holos component. A holos
// component may manage resources in multiple namespaces, in this case
// consider setting the component namespace to default.
//
// This field is optional because not all resources require a namespace,
// particularly CRDs and DeployFiles functionality.
// +optional
Namespace string `json:"namespace,omitempty"`
}

View File

@@ -0,0 +1,11 @@
package v1alpha3
const (
APIVersion = "v1alpha3"
BuildPlanKind = "BuildPlan"
HelmChartKind = "HelmChart"
// ChartDir is the directory name created in the holos component directory to cache a chart.
ChartDir = "vendor"
// ResourcesFile is the file name used to store component output when post-processing with kustomize.
ResourcesFile = "resources.yaml"
)

26
api/core/v1alpha3/doc.go Normal file
View File

@@ -0,0 +1,26 @@
// Package v1alpha3 contains the core API contract between the holos cli and CUE
// configuration code. Platform designers, operators, and software developers
// use this API to write configuration in CUE which `holos` loads. The overall
// shape of the API defines imperative actions `holos` should carry out to
// render the complete yaml that represents a Platform.
//
// [Platform] defines the complete configuration of a platform. With the holos
// reference platform this takes the shape of one management cluster and at
// least two workload cluster. Each cluster has multiple [Component]
// resources applied to it.
//
// Each holos component path, e.g. `components/namespaces` produces exactly one
// [BuildPlan] which in turn contains a set of [Component] kinds.
//
// The primary kinds of [Component] are:
//
// 1. [HelmChart] to render config from a helm chart.
// 2. [KustomizeBuild] to render config from [Kustomize]
// 3. [KubernetesObjects] to render [APIObjects] defined directly in CUE
// configuration.
//
// Note that Holos operates as a data pipeline, so the output of a [HelmChart]
// may be provided to [Kustomize] for post-processing.
package v1alpha3
//go:generate ../../../hack/gendoc

38
api/core/v1alpha3/helm.go Normal file
View File

@@ -0,0 +1,38 @@
package v1alpha3
// HelmChart represents a holos component which wraps around an upstream helm
// chart. Holos orchestrates helm by providing values obtained from CUE,
// renders the output using `helm template`, then post-processes the helm output
// yaml using the general functionality provided by [Component], for
// example [Kustomize] post-rendering and mixing in additional kubernetes api
// objects.
type HelmChart struct {
Component `json:",inline"`
Kind string `json:"kind" cue:"\"HelmChart\""`
// Chart represents a helm chart to manage.
Chart Chart `json:"chart"`
// ValuesContent represents the values.yaml file holos passes to the `helm
// template` command.
ValuesContent string `json:"valuesContent"`
// EnableHooks enables helm hooks when executing the `helm template` command.
EnableHooks bool `json:"enableHooks" cue:"bool | *false"`
}
// Chart represents a helm chart.
type Chart struct {
// Name represents the chart name.
Name string `json:"name"`
// Version represents the chart version.
Version string `json:"version"`
// Release represents the chart release when executing helm template.
Release string `json:"release"`
// Repository represents the repository to fetch the chart from.
Repository Repository `json:"repository,omitempty"`
}
// Repository represents a helm chart repository.
type Repository struct {
Name string `json:"name"`
URL string `json:"url"`
}

View File

@@ -0,0 +1,10 @@
package v1alpha3
const KubernetesObjectsKind = "KubernetesObjects"
// KubernetesObjects represents a [Component] composed of Kubernetes API
// objects provided directly from CUE using [APIObjects].
type KubernetesObjects struct {
Component `json:",inline"`
Kind string `json:"kind" cue:"\"KubernetesObjects\""`
}

View File

@@ -0,0 +1,8 @@
package v1alpha3
// KustomizeBuild represents a [Component] that renders plain yaml files in
// the holos component directory using `kubectl kustomize build`.
type KustomizeBuild struct {
Component `json:",inline"`
Kind string `json:"kind" cue:"\"KustomizeBuild\""`
}

View File

@@ -0,0 +1,44 @@
package v1alpha3
import "google.golang.org/protobuf/types/known/structpb"
// Platform represents a platform to manage. A Platform resource informs holos
// which components to build. The platform resource also acts as a container
// for the platform model form values provided by the PlatformService. The
// primary use case is to collect the cluster names, cluster types, platform
// model, and holos components to build into one resource.
type Platform struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind" cue:"\"Platform\""`
// APIVersion represents the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha3\""`
// Metadata represents data about the object such as the Name.
Metadata PlatformMetadata `json:"metadata"`
// Spec represents the specification.
Spec PlatformSpec `json:"spec"`
}
type PlatformMetadata struct {
// Name represents the Platform name.
Name string `json:"name"`
}
// PlatformSpec represents the specification of a Platform. Think of a platform
// specification as a list of platform components to apply to a list of
// kubernetes clusters combined with the user-specified Platform Model.
type PlatformSpec struct {
// Model represents the platform model holos gets from from the
// PlatformService.GetPlatform rpc method and provides to CUE using a tag.
Model structpb.Struct `json:"model" cue:"{...}"`
// Components represents a list of holos components to manage.
Components []PlatformSpecComponent `json:"components"`
}
// PlatformSpecComponent represents a holos component to build or render.
type PlatformSpecComponent struct {
// Path is the path of the component relative to the platform root.
Path string `json:"path"`
// Cluster is the cluster name to provide when rendering the component.
Cluster string `json:"cluster"`
}

37
api/meta/v1alpha2/meta.go Normal file
View File

@@ -0,0 +1,37 @@
package v1alpha2
// TypeMeta describes an individual object in an API response or request with
// strings representing the type of the object and its API schema version.
// Structures that are versioned or persisted should inline TypeMeta.
type TypeMeta struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind"`
// APIVersion defines the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha2\""`
}
func (tm *TypeMeta) GetKind() string {
return tm.Kind
}
func (tm *TypeMeta) GetAPIVersion() string {
return tm.APIVersion
}
// Discriminator discriminates the kind of an api object.
type Discriminator interface {
// GetKind returns Kind.
GetKind() string
// GetAPIVersion returns APIVersion.
GetAPIVersion() string
}
// ObjectMeta represents metadata of a holos component object. The fields are a
// copy of upstream kubernetes api machinery but are holos objects distinct from
// kubernetes api objects.
type ObjectMeta struct {
// Name uniquely identifies the holos component instance and must be suitable as a file name.
Name string `json:"name,omitempty"`
// Namespace confines a holos component to a single namespace via kustomize if set.
Namespace string `json:"namespace,omitempty"`
}

View File

@@ -0,0 +1,150 @@
// Package v1alpha3 contains CUE definitions intended as convenience wrappers
// around the core data types defined in package core. The purpose of these
// wrappers is to make life easier for platform engineers by reducing boiler
// plate code and generating component build plans in a consistent manner.
package v1alpha3
import (
core "github.com/holos-run/holos/api/core/v1alpha3"
"google.golang.org/protobuf/types/known/structpb"
)
//go:generate ../../../hack/gendoc
// Helm provides a BuildPlan via the Output field which contains one HelmChart
// from package core. Useful as a convenience wrapper to render a HelmChart
// with optional mix-in resources and Kustomization post-processing.
type Helm struct {
// Name represents the chart name.
Name string
// Version represents the chart version.
Version string
// Namespace represents the helm namespace option when rendering the chart.
Namespace string
// Resources are kubernetes api objects to mix into the output.
Resources map[string]any `cue:"{...}"`
// Repo represents the chart repository
Repo struct {
Name string `json:"name"`
URL string `json:"url"`
}
// Values represents data to marshal into a values.yaml for helm.
Values interface{} `cue:"{...}"`
// Chart represents the derived HelmChart for inclusion in the BuildPlan
// Output field value. The default HelmChart field values are derived from
// other Helm field values and should be sufficient for most use cases.
Chart core.HelmChart
// EnableKustomizePostProcessor processes helm output with kustomize if true.
EnableKustomizePostProcessor bool `cue:"true | *false"`
// KustomizeFiles represents additional files to include in a Kustomization
// resources list. Useful to patch helm output. The implementation is a
// struct with filename keys and structs as values. Holos encodes the struct
// value to yaml then writes the result to the filename key. Component
// authors may then reference the filename in the kustomization.yaml resources
// or patches lists.
// Requires EnableKustomizePostProcessor: true.
KustomizeFiles map[string]any `cue:"{[string]: {...}}"`
// KustomizePatches represents patches to apply to the helm output. Requires
// EnableKustomizePostProcessor: true.
KustomizePatches map[core.InternalLabel]any `cue:"{[string]: {...}}"`
// KustomizeResources represents additional resources files to include in the
// kustomize resources list.
KustomizeResources map[string]any `cue:"{[string]: {...}}"`
// ArgoConfig represents the ArgoCD GitOps configuration for this Component.
ArgoConfig ArgoConfig
// Output represents the derived BuildPlan for the Holos cli to render.
Output core.BuildPlan
}
// Resources represents the default schema for a Kubernetes API object resource.
// For example, a Service, Namespace or Deployment. The top level key is the
// kind of resource so default behavior and strict schema enforcement may be
// enforced for the kind. The second level keys are an arbitrary internal
// label, which serves as the default value for the resource metadata name
// field, but may differ for situations where the same resource kind and name
// are managed in different namespaces.
//
// Refer to [definitions.cue] for the CUE schema definition as an example to
// build on when defining your own Components.
//
// [definitions.cue]: https://github.com/holos-run/holos/blob/main/internal/generate/platforms/cue.mod/pkg/github.com/holos-run/holos/api/schema/v1alpha3/definitions.cue#L9
// type Resources map[string]map[string]any
// ArgoConfig represents the ArgoCD GitOps configuration for a Component.
// Useful to define once at the root of the Platform configuration and reuse
// across all Components.
type ArgoConfig struct {
// Enabled causes holos to render an ArgoCD Application resource for GitOps if true.
Enabled bool `cue:"true | *false"`
// ClusterName represents the cluster within the platform the Application
// resource is intended for.
ClusterName string
// DeployRoot represents the path from the git repository root to the `deploy`
// rendering output directory. Used as a prefix for the
// Application.spec.source.path field.
DeployRoot string `cue:"string | *\".\""`
// RepoURL represents the value passed to the Application.spec.source.repoURL
// field.
RepoURL string
// TargetRevision represents the value passed to the
// Application.spec.source.targetRevision field. Defaults to the branch named
// main.
TargetRevision string `cue:"string | *\"main\""`
}
// Cluster represents a cluster managed by the Platform.
type Cluster struct {
// Name represents the cluster name, for example "east1", "west1", or
// "management".
Name string `json:"name"`
// Primary represents if the cluster is marked as the primary among a set of
// candidate clusters. Useful for promotion of database leaders.
Primary bool `json:"primary" cue:"true | *false"`
}
// Fleet represents a named collection of similarly configured Clusters. Useful
// to segregate workload clusters from their management cluster.
type Fleet struct {
Name string `json:"name"`
// Clusters represents a mapping of Clusters by their name.
Clusters map[string]Cluster `json:"clusters" cue:"{[Name=_]: name: Name}"`
}
// StandardFleets represents the standard set of Clusters in a Platform
// segmented into Fleets by their purpose. The management Fleet contains a
// single Cluster, for example a GKE autopilot cluster with no workloads
// deployed for reliability and cost efficiency. The workload Fleet contains
// all other Clusters which contain workloads and sync Secrets from the
// management cluster.
type StandardFleets struct {
// Workload represents a Fleet of zero or more workload Clusters.
Workload Fleet `json:"workload" cue:"{name: \"workload\"}"`
// Management represents a Fleet with one Cluster named management.
Management Fleet `json:"management" cue:"{name: \"management\", clusters: management: _}"`
}
// Platform is a convenience structure to produce a core Platform specification
// value in the Output field. Useful to collect components at the root of the
// Platform configuration tree as a struct, which are automatically converted
// into a list for the core Platform spec output.
type Platform struct {
// Name represents the Platform name.
Name string `cue:"string | *\"holos\""`
// Components is a structured map of components to manage by their name.
Components map[string]core.PlatformSpecComponent
// Model represents the Platform model holos gets from from the
// PlatformService.GetPlatform rpc method and provides to CUE using a tag.
Model structpb.Struct `cue:"{...}"`
// Output represents the core Platform spec for the holos cli to iterate over
// and render each listed Component, injecting the Model.
Output core.Platform
}

View File

@@ -1,6 +1,7 @@
package v1alpha1
import (
"errors"
"fmt"
"strings"
)
@@ -16,6 +17,10 @@ type BuildPlan struct {
type BuildPlanSpec struct {
Disabled bool `json:"disabled,omitempty" yaml:"disabled,omitempty"`
Components BuildPlanComponents `json:"components,omitempty" yaml:"components,omitempty"`
// DeployFiles keys represent file paths relative to the cluster deploy
// directory. Map values represent the string encoded file contents. Used to
// write the argocd Application, but may be used to render any file from CUE.
DeployFiles FileContentMap `json:"deployFiles,omitempty" yaml:"deployFiles,omitempty"`
}
type BuildPlanComponents struct {
@@ -34,7 +39,18 @@ func (bp *BuildPlan) Validate() error {
errs = append(errs, fmt.Sprintf("apiVersion invalid: want: %s have: %s", APIVersion, bp.APIVersion))
}
if len(errs) > 0 {
return fmt.Errorf("invalid BuildPlan: " + strings.Join(errs, ", "))
return errors.New("invalid BuildPlan: " + strings.Join(errs, ", "))
}
return nil
}
func (bp *BuildPlan) ResultCapacity() (count int) {
if bp == nil {
return 0
}
count = len(bp.Spec.Components.HelmChartList) +
len(bp.Spec.Components.KubernetesObjectsList) +
len(bp.Spec.Components.KustomizeBuildList) +
len(bp.Spec.Components.Resources)
return count
}

View File

@@ -20,3 +20,11 @@ type HolosComponent struct {
func (hc *HolosComponent) NewResult() *Result {
return &Result{HolosComponent: *hc}
}
func (hc *HolosComponent) GetAPIVersion() string {
return hc.APIVersion
}
func (hc *HolosComponent) GetKind() string {
return hc.Kind
}

13
api/v1alpha1/form.go Normal file
View File

@@ -0,0 +1,13 @@
package v1alpha1
import object "github.com/holos-run/holos/service/gen/holos/object/v1alpha1"
// Form represents a collection of Formly json powered form.
type Form struct {
TypeMeta `json:",inline" yaml:",inline"`
Spec FormSpec `json:"spec" yaml:"spec"`
}
type FormSpec struct {
Form object.Form `json:"form" yaml:"form"`
}

View File

@@ -6,11 +6,12 @@ import (
"os"
"path/filepath"
"strings"
"syscall"
"github.com/holos-run/holos"
"github.com/holos-run/holos/pkg/logger"
"github.com/holos-run/holos/pkg/util"
"github.com/holos-run/holos/pkg/wrapper"
"github.com/holos-run/holos/internal/errors"
"github.com/holos-run/holos/internal/logger"
"github.com/holos-run/holos/internal/util"
)
// A HelmChart represents a helm command to provide chart values in order to render kubernetes api objects.
@@ -42,7 +43,7 @@ func (hc *HelmChart) Render(ctx context.Context, path holos.InstancePath) (*Resu
}
result.addObjectMap(ctx, hc.APIObjectMap)
if err := result.kustomize(ctx); err != nil {
return nil, wrapper.Wrap(fmt.Errorf("could not kustomize: %w", err))
return nil, errors.Wrap(fmt.Errorf("could not kustomize: %w", err))
}
return &result, nil
}
@@ -64,13 +65,13 @@ func (hc *HelmChart) helm(ctx context.Context, r *Result, path holos.InstancePat
out, err := util.RunCmd(ctx, "helm", "repo", "add", repo.Name, repo.URL)
if err != nil {
log.ErrorContext(ctx, "could not run helm", "stderr", out.Stderr.String(), "stdout", out.Stdout.String())
return wrapper.Wrap(fmt.Errorf("could not run helm repo add: %w", err))
return errors.Wrap(fmt.Errorf("could not run helm repo add: %w", err))
}
// Update repository
out, err = util.RunCmd(ctx, "helm", "repo", "update", repo.Name)
if err != nil {
log.ErrorContext(ctx, "could not run helm", "stderr", out.Stderr.String(), "stdout", out.Stdout.String())
return wrapper.Wrap(fmt.Errorf("could not run helm repo update: %w", err))
return errors.Wrap(fmt.Errorf("could not run helm repo update: %w", err))
}
} else {
log.DebugContext(ctx, "no chart repository url proceeding assuming oci chart")
@@ -85,13 +86,13 @@ func (hc *HelmChart) helm(ctx context.Context, r *Result, path holos.InstancePat
// Write values file
tempDir, err := os.MkdirTemp("", "holos")
if err != nil {
return wrapper.Wrap(fmt.Errorf("could not make temp dir: %w", err))
return errors.Wrap(fmt.Errorf("could not make temp dir: %w", err))
}
defer util.Remove(ctx, tempDir)
valuesPath := filepath.Join(tempDir, "values.yaml")
if err := os.WriteFile(valuesPath, []byte(hc.ValuesContent), 0644); err != nil {
return wrapper.Wrap(fmt.Errorf("could not write values: %w", err))
return errors.Wrap(fmt.Errorf("could not write values: %w", err))
}
log.DebugContext(ctx, "helm: wrote values", "path", valuesPath, "bytes", len(hc.ValuesContent))
@@ -112,7 +113,7 @@ func (hc *HelmChart) helm(ctx context.Context, r *Result, path holos.InstancePat
err = fmt.Errorf("%s: %w", line, err)
}
}
return wrapper.Wrap(fmt.Errorf("could not run helm template: %w", err))
return errors.Wrap(fmt.Errorf("could not run helm template: %w", err))
}
r.accumulatedOutput = helmOut.Stdout.String()
@@ -121,12 +122,20 @@ func (hc *HelmChart) helm(ctx context.Context, r *Result, path holos.InstancePat
}
// cacheChart stores a cached copy of Chart in the chart subdirectory of path.
//
// It is assumed that the only method responsible for writing to chartDir is
// cacheChart itself.
//
// This relies on the atomicity of moving temporary directories into place on
// the same filesystem via os.Rename. If a syscall.EEXIST error occurs during
// renaming, it indicates that the cached chart already exists, which is an
// expected scenario when this function is called concurrently.
func cacheChart(ctx context.Context, path holos.InstancePath, chartDir string, chart Chart) error {
log := logger.FromContext(ctx)
cacheTemp, err := os.MkdirTemp(string(path), chartDir)
if err != nil {
return wrapper.Wrap(fmt.Errorf("could not make temp dir: %w", err))
return errors.Wrap(fmt.Errorf("could not make temp dir: %w", err))
}
defer util.Remove(ctx, cacheTemp)
@@ -136,15 +145,36 @@ func cacheChart(ctx context.Context, path holos.InstancePath, chartDir string, c
}
helmOut, err := util.RunCmd(ctx, "helm", "pull", "--destination", cacheTemp, "--untar=true", "--version", chart.Version, chartName)
if err != nil {
return wrapper.Wrap(fmt.Errorf("could not run helm pull: %w", err))
return errors.Wrap(fmt.Errorf("could not run helm pull: %w", err))
}
log.Debug("helm pull", "stdout", helmOut.Stdout, "stderr", helmOut.Stderr)
cachePath := filepath.Join(string(path), chartDir)
if err := os.Rename(cacheTemp, cachePath); err != nil {
return wrapper.Wrap(fmt.Errorf("could not rename: %w", err))
if err := os.MkdirAll(cachePath, 0777); err != nil {
return errors.Wrap(fmt.Errorf("could not mkdir: %w", err))
}
log.InfoContext(ctx, "cached", "chart", chart.Name, "version", chart.Version, "path", cachePath)
items, err := os.ReadDir(cacheTemp)
if err != nil {
return errors.Wrap(fmt.Errorf("could not read directory: %w", err))
}
for _, item := range items {
src := filepath.Join(cacheTemp, item.Name())
dst := filepath.Join(cachePath, item.Name())
log.DebugContext(ctx, "rename", "src", src, "dst", dst)
if err := os.Rename(src, dst); err != nil {
var linkErr *os.LinkError
if errors.As(err, &linkErr) && errors.Is(linkErr.Err, syscall.EEXIST) {
log.DebugContext(ctx, "cache already exists", "chart", chart.Name, "chart_version", chart.Version, "path", cachePath)
} else {
return errors.Wrap(fmt.Errorf("could not rename: %w", err))
}
}
}
log.InfoContext(ctx, "cached", "chart", chart.Name, "chart_version", chart.Version, "path", cachePath)
return nil
}

View File

@@ -4,9 +4,9 @@ import (
"context"
"github.com/holos-run/holos"
"github.com/holos-run/holos/pkg/logger"
"github.com/holos-run/holos/pkg/util"
"github.com/holos-run/holos/pkg/wrapper"
"github.com/holos-run/holos/internal/errors"
"github.com/holos-run/holos/internal/logger"
"github.com/holos-run/holos/internal/util"
)
const KustomizeBuildKind = "KustomizeBuild"
@@ -37,7 +37,7 @@ func (kb *KustomizeBuild) Render(ctx context.Context, path holos.InstancePath) (
kOut, err := util.RunCmd(ctx, "kubectl", "kustomize", string(path))
if err != nil {
log.ErrorContext(ctx, kOut.Stderr.String())
return nil, wrapper.Wrap(err)
return nil, errors.Wrap(err)
}
// Replace the accumulated output
result.accumulatedOutput = kOut.Stdout.String()

32
api/v1alpha1/platform.go Normal file
View File

@@ -0,0 +1,32 @@
package v1alpha1
import "google.golang.org/protobuf/types/known/structpb"
// Platform represents a platform to manage. A Platform resource informs holos
// which components to build. The platform resource also acts as a container
// for the platform model form values provided by the PlatformService. The
// primary use case is to collect the cluster names, cluster types, platform
// model, and holos components to build into one resource.
type Platform struct {
TypeMeta `json:",inline" yaml:",inline"`
Metadata ObjectMeta `json:"metadata" yaml:"metadata"`
Spec PlatformSpec `json:"spec" yaml:"spec"`
}
// PlatformSpec represents the platform build plan specification.
type PlatformSpec struct {
// Model represents the platform model holos gets from from the
// holos.platform.v1alpha1.PlatformService.GetPlatform method and provides to
// CUE using a tag.
Model structpb.Struct `json:"model" yaml:"model"`
Components []PlatformSpecComponent `json:"components" yaml:"components"`
}
// PlatformSpecComponent represents a component to build or render with flags to
// pass, for example the cluster name.
type PlatformSpecComponent struct {
// Path is the path of the component relative to the platform root.
Path string `json:"path" yaml:"path"`
// Cluster is the cluster name to use when building the component.
Cluster string `json:"cluster" yaml:"cluster"`
}

View File

@@ -7,9 +7,9 @@ import (
"path/filepath"
"slices"
"github.com/holos-run/holos/pkg/logger"
"github.com/holos-run/holos/pkg/util"
"github.com/holos-run/holos/pkg/wrapper"
"github.com/holos-run/holos/internal/errors"
"github.com/holos-run/holos/internal/logger"
"github.com/holos-run/holos/internal/util"
)
// Result is the build result for display or writing. Holos components Render the Result as a data pipeline.
@@ -17,6 +17,18 @@ type Result struct {
HolosComponent
// accumulatedOutput accumulates rendered api objects.
accumulatedOutput string
// DeployFiles keys represent file paths relative to the cluster deploy
// directory. Map values represent the string encoded file contents. Used to
// write the argocd Application, but may be used to render any file from CUE.
DeployFiles FileContentMap `json:"deployFiles,omitempty" yaml:"deployFiles,omitempty"`
}
// Continue returns true if Skip is true indicating the result is to be skipped over.
func (r *Result) Continue() bool {
if r == nil {
return false
}
return r.Skip
}
func (r *Result) Name() string {
@@ -82,7 +94,7 @@ func (r *Result) kustomize(ctx context.Context) error {
}
tempDir, err := os.MkdirTemp("", "holos.kustomize")
if err != nil {
return wrapper.Wrap(err)
return errors.Wrap(err)
}
defer util.Remove(ctx, tempDir)
@@ -91,7 +103,7 @@ func (r *Result) kustomize(ctx context.Context) error {
b := []byte(r.AccumulatedOutput())
b = util.EnsureNewline(b)
if err := os.WriteFile(target, b, 0644); err != nil {
return wrapper.Wrap(fmt.Errorf("could not write resources: %w", err))
return errors.Wrap(fmt.Errorf("could not write resources: %w", err))
}
log.DebugContext(ctx, "wrote: "+target, "op", "write", "path", target, "bytes", len(b))
@@ -99,12 +111,12 @@ func (r *Result) kustomize(ctx context.Context) error {
for file, content := range r.KustomizeFiles {
target := filepath.Join(tempDir, file)
if err := os.MkdirAll(filepath.Dir(target), 0755); err != nil {
return wrapper.Wrap(err)
return errors.Wrap(err)
}
b := []byte(content)
b = util.EnsureNewline(b)
if err := os.WriteFile(target, b, 0644); err != nil {
return wrapper.Wrap(fmt.Errorf("could not write: %w", err))
return errors.Wrap(fmt.Errorf("could not write: %w", err))
}
log.DebugContext(ctx, "wrote: "+target, "op", "write", "path", target, "bytes", len(b))
}
@@ -113,25 +125,40 @@ func (r *Result) kustomize(ctx context.Context) error {
kOut, err := util.RunCmd(ctx, "kubectl", "kustomize", tempDir)
if err != nil {
log.ErrorContext(ctx, kOut.Stderr.String())
return wrapper.Wrap(err)
return errors.Wrap(err)
}
// Replace the accumulated output
r.accumulatedOutput = kOut.Stdout.String()
return nil
}
func (r *Result) WriteDeployFiles(ctx context.Context, path string) error {
log := logger.FromContext(ctx)
if len(r.DeployFiles) == 0 {
return nil
}
for k, content := range r.DeployFiles {
path := filepath.Join(path, k)
if err := r.Save(ctx, path, content); err != nil {
return errors.Wrap(err)
}
log.InfoContext(ctx, "wrote deploy file", "path", path, "bytes", len(content))
}
return nil
}
// Save writes the content to the filesystem for git ops.
func (r *Result) Save(ctx context.Context, path string, content string) error {
log := logger.FromContext(ctx)
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, os.FileMode(0775)); err != nil {
log.WarnContext(ctx, "could not mkdir", "path", dir, "err", err)
return wrapper.Wrap(err)
return errors.Wrap(err)
}
// Write the kube api objects
// Write the file content
if err := os.WriteFile(path, []byte(content), os.FileMode(0644)); err != nil {
log.WarnContext(ctx, "could not write", "path", path, "err", err)
return wrapper.Wrap(err)
return errors.Wrap(err)
}
log.DebugContext(ctx, "out: wrote "+path, "action", "write", "path", path, "status", "ok")
return nil

View File

@@ -8,3 +8,13 @@ type TypeMeta struct {
func (tm *TypeMeta) GetKind() string {
return tm.Kind
}
func (tm *TypeMeta) GetAPIVersion() string {
return tm.APIVersion
}
// Discriminator is an interface to discriminate the kind api object.
type Discriminator interface {
GetKind() string
GetAPIVersion() string
}

20
buf.gen.yaml Normal file
View File

@@ -0,0 +1,20 @@
# Generates gRPC and ConnectRPC bindings for Go and TypeScript
#
# Note: protoc-gen-connect-query is the primary method of wiring up the React
# frontend.
version: v1
plugins:
- plugin: go
out: service/gen
opt: paths=source_relative
- plugin: connect-go
out: service/gen
opt: paths=source_relative
- plugin: es
out: internal/frontend/holos/src/app/gen
opt:
- target=ts
- plugin: connect-es
out: internal/frontend/holos/src/app/gen
opt:
- target=ts

8
buf.lock Normal file
View File

@@ -0,0 +1,8 @@
# Generated by buf. DO NOT EDIT.
version: v1
deps:
- remote: buf.build
owner: bufbuild
repository: protovalidate
commit: b983156c5e994cc9892e0ce3e64e17e0
digest: shake256:fb47a62989d38c2529bcc5cd86ded43d800eb84cee82b42b9e8a9e815d4ee8134a0fb9d0ce8299b27c2d2bbb7d6ade0c4ad5a8a4d467e1e2c7ca619ae9f634e2

3
buf.work.yaml Normal file
View File

@@ -0,0 +1,3 @@
version: v1
directories:
- service

View File

@@ -1,8 +1,9 @@
package main
import (
"github.com/holos-run/holos/pkg/cli"
"os"
"github.com/holos-run/holos/internal/cli"
)
func main() {

View File

@@ -1,10 +1,11 @@
package main
import (
"github.com/holos-run/holos/pkg/cli"
"github.com/rogpeppe/go-internal/testscript"
"os"
"testing"
"github.com/holos-run/holos/internal/cli"
"github.com/rogpeppe/go-internal/testscript"
)
func TestMain(m *testing.M) {

View File

@@ -2,6 +2,8 @@
exec holos build ./foo/... --log-level debug
stdout '^bf2bc7f9-9ba0-4f9e-9bd2-9a205627eb0b$'
-- platform.config.json --
{}
-- cue.mod --
package holos
-- foo/constraints.cue --
@@ -20,6 +22,7 @@ spec: components: KubernetesObjectsList: [
package holos
_cluster: string @tag(cluster, string)
_platform_config: string @tag(platform_config, string)
#KubernetesObjects: {
apiVersion: "holos.run/v1alpha1"

View File

@@ -3,12 +3,15 @@
stderr 'apiObjectMap.foo.bar: cannot convert incomplete value'
stderr '/component.cue:\d+:\d+$'
-- platform.config.json --
{}
-- cue.mod --
package holos
-- component.cue --
package holos
_cluster: string @tag(cluster, string)
_platform_config: string @tag(platform_config, string)
apiVersion: "holos.run/v1alpha1"
kind: "BuildPlan"

View File

@@ -3,6 +3,8 @@ exec holos build .
stdout '^kind: SecretStore$'
stdout '# Source: CUE apiObjects.SecretStore.default'
-- platform.config.json --
{}
-- cue.mod --
package holos
-- component.cue --
@@ -13,6 +15,7 @@ kind: "BuildPlan"
spec: components: KubernetesObjectsList: [{apiObjectMap: #APIObjects.apiObjectMap}]
_cluster: string @tag(cluster, string)
_platform_config: string @tag(platform_config, string)
#SecretStore: {
kind: string

View File

@@ -4,6 +4,8 @@ stdout '^kind: SecretStore$'
stdout '# Source: CUE apiObjects.SecretStore.default'
stderr 'skipping helm: no chart name specified'
-- platform.config.json --
{}
-- cue.mod --
package holos
-- component.cue --
@@ -14,6 +16,7 @@ kind: "BuildPlan"
spec: components: HelmChartList: [{apiObjectMap: #APIObjects.apiObjectMap}]
_cluster: string @tag(cluster, string)
_platform_config: string @tag(platform_config, string)
#SecretStore: {
kind: string

View File

@@ -2,6 +2,8 @@
! exec holos build .
stderr 'apiObjects.secretstore.default.foo: field not allowed'
-- platform.config.json --
{}
-- cue.mod --
package holos
-- component.cue --
@@ -10,6 +12,7 @@ package holos
apiVersion: "holos.run/v1alpha1"
kind: "KubernetesObjects"
cluster: string @tag(cluster, string)
_platform_config: string @tag(platform_config, string)
#SecretStore: {
metadata: name: string

View File

@@ -2,6 +2,8 @@
! exec holos build .
stderr 'Error: execution error at \(zitadel/templates/secret_zitadel-masterkey.yaml:2:4\): Either set .Values.zitadel.masterkey xor .Values.zitadel.masterkeySecretName'
-- platform.config.json --
{}
-- cue.mod --
package holos
-- zitadel.cue --
@@ -12,6 +14,7 @@ kind: "BuildPlan"
spec: components: HelmChartList: [_HelmChart]
_cluster: string @tag(cluster, string)
_platform_config: string @tag(platform_config, string)
_HelmChart: {
apiVersion: "holos.run/v1alpha1"

View File

@@ -1,15 +1,18 @@
# Kustomize is a supported holos component kind
exec holos render --cluster-name=mycluster . --log-level=debug
exec holos render component --cluster-name=mycluster . --log-level=debug
# Want generated output
cmp want.yaml deploy/clusters/mycluster/components/kstest/kstest.gen.yaml
-- platform.config.json --
{}
-- cue.mod --
package holos
-- component.cue --
package holos
_cluster: string @tag(cluster, string)
_platform_config: string @tag(platform_config, string)
apiVersion: "holos.run/v1alpha1"
kind: "BuildPlan"

View File

@@ -3,11 +3,14 @@
! exec holos build .
stderr 'unknown field \\"TypoKubernetesObjectsList\\"'
-- platform.config.json --
{}
-- cue.mod --
package holos
-- component.cue --
package holos
_cluster: string @tag(cluster, string)
_platform_config: string @tag(platform_config, string)
apiVersion: "holos.run/v1alpha1"
kind: "BuildPlan"

View File

@@ -1,5 +1,3 @@
exec holos --version
# want version with no v on stdout
stdout -count=1 '^\d+\.\d+\.\d+$'
# want nothing on stderr
! stderr .

View File

@@ -1 +0,0 @@
module: "github.com/holos-run/holos"

3
doc/md/api/core/index.md Normal file
View File

@@ -0,0 +1,3 @@
# Core API
- [v1alpha2](v1alpha2)

403
doc/md/api/core/v1alpha2.md Normal file
View File

@@ -0,0 +1,403 @@
<!-- Code generated by gomarkdoc. DO NOT EDIT -->
# v1alpha2
```go
import "github.com/holos-run/holos/api/core/v1alpha2"
```
Package v1alpha2 contains the core API contract between the holos cli and CUE configuration code. Platform designers, operators, and software developers use this API to write configuration in CUE which \`holos\` loads. The overall shape of the API defines imperative actions \`holos\` should carry out to render the complete yaml that represents a Platform.
[Platform](<#Platform>) defines the complete configuration of a platform. With the holos reference platform this takes the shape of one management cluster and at least two workload cluster. Each cluster has multiple [HolosComponent](<#HolosComponent>) resources applied to it.
Each holos component path, e.g. \`components/namespaces\` produces exactly one [BuildPlan](<#BuildPlan>) which in turn contains a set of [HolosComponent](<#HolosComponent>) kinds.
The primary kinds of [HolosComponent](<#HolosComponent>) are:
1. [HelmChart](<#HelmChart>) to render config from a helm chart.
2. [KustomizeBuild](<#KustomizeBuild>) to render config from [Kustomize](<#Kustomize>)
3. [KubernetesObjects](<#KubernetesObjects>) to render [APIObjects](<#APIObjects>) defined directly in CUE configuration.
Note that Holos operates as a data pipeline, so the output of a [HelmChart](<#HelmChart>) may be provided to [Kustomize](<#Kustomize>) for post\-processing.
## Index
- [Constants](<#constants>)
- [type APIObject](<#APIObject>)
- [type APIObjectMap](<#APIObjectMap>)
- [type APIObjects](<#APIObjects>)
- [type BuildPlan](<#BuildPlan>)
- [type BuildPlanComponents](<#BuildPlanComponents>)
- [type BuildPlanSpec](<#BuildPlanSpec>)
- [type Chart](<#Chart>)
- [type FileContent](<#FileContent>)
- [type FileContentMap](<#FileContentMap>)
- [type FilePath](<#FilePath>)
- [type HelmChart](<#HelmChart>)
- [type HolosComponent](<#HolosComponent>)
- [type Kind](<#Kind>)
- [type KubernetesObjects](<#KubernetesObjects>)
- [type Kustomize](<#Kustomize>)
- [type KustomizeBuild](<#KustomizeBuild>)
- [type Label](<#Label>)
- [type Metadata](<#Metadata>)
- [type Platform](<#Platform>)
- [type PlatformMetadata](<#PlatformMetadata>)
- [type PlatformSpec](<#PlatformSpec>)
- [type PlatformSpecComponent](<#PlatformSpecComponent>)
- [type Repository](<#Repository>)
## Constants
<a name="APIVersion"></a>
```go
const (
APIVersion = "v1alpha2"
BuildPlanKind = "BuildPlan"
HelmChartKind = "HelmChart"
// ChartDir is the directory name created in the holos component directory to cache a chart.
ChartDir = "vendor"
// ResourcesFile is the file name used to store component output when post-processing with kustomize.
ResourcesFile = "resources.yaml"
)
```
<a name="KubernetesObjectsKind"></a>
```go
const KubernetesObjectsKind = "KubernetesObjects"
```
<a name="APIObject"></a>
## type APIObject {#APIObject}
APIObject represents the most basic generic form of a single kubernetes api object. Represented as a JSON object internally for compatibility between tools, for example loading from CUE.
```go
type APIObject structpb.Struct
```
<a name="APIObjectMap"></a>
## type APIObjectMap {#APIObjectMap}
APIObjectMap represents the marshalled yaml representation of kubernetes api objects. Do not produce an APIObjectMap directly, instead use [APIObjects](<#APIObjects>) to produce the marshalled yaml representation from CUE data, then provide the result to [HolosComponent](<#HolosComponent>).
```go
type APIObjectMap map[Kind]map[Label]string
```
<a name="APIObjects"></a>
## type APIObjects {#APIObjects}
APIObjects represents Kubernetes API objects defined directly from CUE code. Useful to mix in resources to any kind of [HolosComponent](<#HolosComponent>), for example adding an ExternalSecret resource to a [HelmChart](<#HelmChart>).
[Kind](<#Kind>) must be the resource kind, e.g. Deployment or Service.
[Label](<#Label>) is an arbitrary internal identifier to uniquely identify the resource within the context of a \`holos\` command. Holos will never write the intermediate label to rendered output.
Refer to [HolosComponent](<#HolosComponent>) which accepts an [APIObjectMap](<#APIObjectMap>) field provided by [APIObjects](<#APIObjects>).
```go
type APIObjects struct {
APIObjects map[Kind]map[Label]APIObject `json:"apiObjects"`
APIObjectMap APIObjectMap `json:"apiObjectMap"`
}
```
<a name="BuildPlan"></a>
## type BuildPlan {#BuildPlan}
BuildPlan represents a build plan for the holos cli to execute. The purpose of a BuildPlan is to define one or more [HolosComponent](<#HolosComponent>) kinds. For example a [HelmChart](<#HelmChart>), [KustomizeBuild](<#KustomizeBuild>), or [KubernetesObjects](<#KubernetesObjects>).
A BuildPlan usually has an additional empty [KubernetesObjects](<#KubernetesObjects>) for the purpose of using the [HolosComponent](<#HolosComponent>) DeployFiles field to deploy an ArgoCD or Flux gitops resource for the holos component.
```go
type BuildPlan struct {
Kind string `json:"kind" cue:"\"BuildPlan\""`
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha2\""`
Spec BuildPlanSpec `json:"spec"`
}
```
<a name="BuildPlanComponents"></a>
## type BuildPlanComponents {#BuildPlanComponents}
```go
type BuildPlanComponents struct {
Resources map[Label]KubernetesObjects `json:"resources,omitempty"`
KubernetesObjectsList []KubernetesObjects `json:"kubernetesObjectsList,omitempty"`
HelmChartList []HelmChart `json:"helmChartList,omitempty"`
KustomizeBuildList []KustomizeBuild `json:"kustomizeBuildList,omitempty"`
}
```
<a name="BuildPlanSpec"></a>
## type BuildPlanSpec {#BuildPlanSpec}
BuildPlanSpec represents the specification of the build plan.
```go
type BuildPlanSpec struct {
// Disabled causes the holos cli to take no action over the [BuildPlan].
Disabled bool `json:"disabled,omitempty"`
// Components represents multiple [HolosComponent] kinds to manage.
Components BuildPlanComponents `json:"components,omitempty"`
}
```
<a name="Chart"></a>
## type Chart {#Chart}
Chart represents a helm chart.
```go
type Chart struct {
// Name represents the chart name.
Name string `json:"name"`
// Version represents the chart version.
Version string `json:"version"`
// Release represents the chart release when executing helm template.
Release string `json:"release"`
// Repository represents the repository to fetch the chart from.
Repository Repository `json:"repository,omitempty"`
}
```
<a name="FileContent"></a>
## type FileContent {#FileContent}
FileContent represents file contents.
```go
type FileContent string
```
<a name="FileContentMap"></a>
## type FileContentMap {#FileContentMap}
FileContentMap represents a mapping of file paths to file contents. Paths are relative to the \`holos\` output "deploy" directory, and may contain sub\-directories.
```go
type FileContentMap map[FilePath]FileContent
```
<a name="FilePath"></a>
## type FilePath {#FilePath}
FilePath represents a file path.
```go
type FilePath string
```
<a name="HelmChart"></a>
## type HelmChart {#HelmChart}
HelmChart represents a holos component which wraps around an upstream helm chart. Holos orchestrates helm by providing values obtained from CUE, renders the output using \`helm template\`, then post\-processes the helm output yaml using the general functionality provided by [HolosComponent](<#HolosComponent>), for example [Kustomize](<#Kustomize>) post\-rendering and mixing in additional kubernetes api objects.
```go
type HelmChart struct {
HolosComponent `json:",inline"`
Kind string `json:"kind" cue:"\"HelmChart\""`
// Chart represents a helm chart to manage.
Chart Chart `json:"chart"`
// ValuesContent represents the values.yaml file holos passes to the `helm
// template` command.
ValuesContent string `json:"valuesContent"`
// EnableHooks enables helm hooks when executing the `helm template` command.
EnableHooks bool `json:"enableHooks" cue:"bool | *false"`
}
```
<a name="HolosComponent"></a>
## type HolosComponent {#HolosComponent}
HolosComponent defines the fields common to all holos component kinds. Every holos component kind should embed HolosComponent.
```go
type HolosComponent struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind"`
// APIVersion represents the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha2\""`
// Metadata represents data about the holos component such as the Name.
Metadata Metadata `json:"metadata"`
// APIObjectMap holds the marshalled representation of api objects. Useful to
// mix in resources to each HolosComponent type, for example adding an
// ExternalSecret to a HelmChart HolosComponent. Refer to [APIObjects].
APIObjectMap APIObjectMap `json:"apiObjectMap,omitempty"`
// DeployFiles represents file paths relative to the cluster deploy directory
// with the value representing the file content. Intended for defining the
// ArgoCD Application resource or Flux Kustomization resource from within CUE,
// but may be used to render any file related to the build plan from CUE.
DeployFiles FileContentMap `json:"deployFiles,omitempty"`
// Kustomize represents a kubectl kustomize build post-processing step.
Kustomize `json:"kustomize,omitempty"`
// Skip causes holos to take no action regarding this component.
Skip bool `json:"skip" cue:"bool | *false"`
}
```
<a name="Kind"></a>
## type Kind {#Kind}
Kind is a kubernetes api object kind. Defined as a type for clarity and type checking.
```go
type Kind string
```
<a name="KubernetesObjects"></a>
## type KubernetesObjects {#KubernetesObjects}
KubernetesObjects represents a [HolosComponent](<#HolosComponent>) composed of Kubernetes API objects provided directly from CUE using [APIObjects](<#APIObjects>).
```go
type KubernetesObjects struct {
HolosComponent `json:",inline"`
Kind string `json:"kind" cue:"\"KubernetesObjects\""`
}
```
<a name="Kustomize"></a>
## type Kustomize {#Kustomize}
Kustomize represents resources necessary to execute a kustomize build. Intended for at least two use cases:
1. Process a [KustomizeBuild](<#KustomizeBuild>) [HolosComponent](<#HolosComponent>) which represents raw yaml file resources in a holos component directory.
2. Post process a [HelmChart](<#HelmChart>) [HolosComponent](<#HolosComponent>) to inject istio, patch jobs, add custom labels, etc...
```go
type Kustomize struct {
// KustomizeFiles holds file contents for kustomize, e.g. patch files.
KustomizeFiles FileContentMap `json:"kustomizeFiles,omitempty"`
// ResourcesFile is the file name used for api objects in kustomization.yaml
ResourcesFile string `json:"resourcesFile,omitempty"`
}
```
<a name="KustomizeBuild"></a>
## type KustomizeBuild {#KustomizeBuild}
KustomizeBuild represents a [HolosComponent](<#HolosComponent>) that renders plain yaml files in the holos component directory using \`kubectl kustomize build\`.
```go
type KustomizeBuild struct {
HolosComponent `json:",inline"`
Kind string `json:"kind" cue:"\"KustomizeBuild\""`
}
```
<a name="Label"></a>
## type Label {#Label}
Label is an arbitrary unique identifier internal to holos itself. The holos cli is expected to never write a Label value to rendered output files, therefore use a [Label](<#Label>) then the identifier must be unique and internal. Defined as a type for clarity and type checking.
A Label is useful to convert a CUE struct to a list, for example producing a list of [APIObject](<#APIObject>) resources from an [APIObjectMap](<#APIObjectMap>). A CUE struct using Label keys is guaranteed to not lose data when rendering output because a Label is expected to never be written to the final output.
```go
type Label string
```
<a name="Metadata"></a>
## type Metadata {#Metadata}
Metadata represents data about the holos component such as the Name.
```go
type Metadata struct {
// Name represents the name of the holos component.
Name string `json:"name"`
// Namespace is the primary namespace of the holos component. A holos
// component may manage resources in multiple namespaces, in this case
// consider setting the component namespace to default.
//
// This field is optional because not all resources require a namespace,
// particularly CRD's and DeployFiles functionality.
// +optional
Namespace string `json:"namespace,omitempty"`
}
```
<a name="Platform"></a>
## type Platform {#Platform}
Platform represents a platform to manage. A Platform resource informs holos which components to build. The platform resource also acts as a container for the platform model form values provided by the PlatformService. The primary use case is to collect the cluster names, cluster types, platform model, and holos components to build into one resource.
```go
type Platform struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind" cue:"\"Platform\""`
// APIVersion represents the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha2\""`
// Metadata represents data about the object such as the Name.
Metadata PlatformMetadata `json:"metadata"`
// Spec represents the specification.
Spec PlatformSpec `json:"spec"`
}
```
<a name="PlatformMetadata"></a>
## type PlatformMetadata {#PlatformMetadata}
```go
type PlatformMetadata struct {
// Name represents the Platform name.
Name string `json:"name"`
}
```
<a name="PlatformSpec"></a>
## type PlatformSpec {#PlatformSpec}
PlatformSpec represents the specification of a Platform. Think of a platform specification as a list of platform components to apply to a list of kubernetes clusters combined with the user\-specified Platform Model.
```go
type PlatformSpec struct {
// Model represents the platform model holos gets from from the
// PlatformService.GetPlatform rpc method and provides to CUE using a tag.
Model structpb.Struct `json:"model"`
// Components represents a list of holos components to manage.
Components []PlatformSpecComponent `json:"components"`
}
```
<a name="PlatformSpecComponent"></a>
## type PlatformSpecComponent {#PlatformSpecComponent}
PlatformSpecComponent represents a holos component to build or render.
```go
type PlatformSpecComponent struct {
// Path is the path of the component relative to the platform root.
Path string `json:"path"`
// Cluster is the cluster name to provide when rendering the component.
Cluster string `json:"cluster"`
}
```
<a name="Repository"></a>
## type Repository {#Repository}
Repository represents a helm chart repository.
```go
type Repository struct {
Name string `json:"name"`
URL string `json:"url"`
}
```
Generated by [gomarkdoc](<https://github.com/princjef/gomarkdoc>)

403
doc/md/api/core/v1alpha3.md Normal file
View File

@@ -0,0 +1,403 @@
<!-- Code generated by gomarkdoc. DO NOT EDIT -->
# v1alpha3
```go
import "github.com/holos-run/holos/api/core/v1alpha3"
```
Package v1alpha3 contains the core API contract between the holos cli and CUE configuration code. Platform designers, operators, and software developers use this API to write configuration in CUE which \`holos\` loads. The overall shape of the API defines imperative actions \`holos\` should carry out to render the complete yaml that represents a Platform.
[Platform](<#Platform>) defines the complete configuration of a platform. With the holos reference platform this takes the shape of one management cluster and at least two workload cluster. Each cluster has multiple [Component](<#Component>) resources applied to it.
Each holos component path, e.g. \`components/namespaces\` produces exactly one [BuildPlan](<#BuildPlan>) which in turn contains a set of [Component](<#Component>) kinds.
The primary kinds of [Component](<#Component>) are:
1. [HelmChart](<#HelmChart>) to render config from a helm chart.
2. [KustomizeBuild](<#KustomizeBuild>) to render config from [Kustomize](<#Kustomize>)
3. [KubernetesObjects](<#KubernetesObjects>) to render [APIObjects](<#APIObjects>) defined directly in CUE configuration.
Note that Holos operates as a data pipeline, so the output of a [HelmChart](<#HelmChart>) may be provided to [Kustomize](<#Kustomize>) for post\-processing.
## Index
- [Constants](<#constants>)
- [type APIObject](<#APIObject>)
- [type APIObjectMap](<#APIObjectMap>)
- [type APIObjects](<#APIObjects>)
- [type BuildPlan](<#BuildPlan>)
- [type BuildPlanComponents](<#BuildPlanComponents>)
- [type BuildPlanSpec](<#BuildPlanSpec>)
- [type Chart](<#Chart>)
- [type Component](<#Component>)
- [type FileContent](<#FileContent>)
- [type FileContentMap](<#FileContentMap>)
- [type FilePath](<#FilePath>)
- [type HelmChart](<#HelmChart>)
- [type InternalLabel](<#InternalLabel>)
- [type Kind](<#Kind>)
- [type KubernetesObjects](<#KubernetesObjects>)
- [type Kustomize](<#Kustomize>)
- [type KustomizeBuild](<#KustomizeBuild>)
- [type Metadata](<#Metadata>)
- [type Platform](<#Platform>)
- [type PlatformMetadata](<#PlatformMetadata>)
- [type PlatformSpec](<#PlatformSpec>)
- [type PlatformSpecComponent](<#PlatformSpecComponent>)
- [type Repository](<#Repository>)
## Constants
<a name="APIVersion"></a>
```go
const (
APIVersion = "v1alpha3"
BuildPlanKind = "BuildPlan"
HelmChartKind = "HelmChart"
// ChartDir is the directory name created in the holos component directory to cache a chart.
ChartDir = "vendor"
// ResourcesFile is the file name used to store component output when post-processing with kustomize.
ResourcesFile = "resources.yaml"
)
```
<a name="KubernetesObjectsKind"></a>
```go
const KubernetesObjectsKind = "KubernetesObjects"
```
<a name="APIObject"></a>
## type APIObject {#APIObject}
APIObject represents the most basic generic form of a single kubernetes api object. Represented as a JSON object internally for compatibility between tools, for example loading from CUE.
```go
type APIObject structpb.Struct
```
<a name="APIObjectMap"></a>
## type APIObjectMap {#APIObjectMap}
APIObjectMap represents the marshalled yaml representation of kubernetes api objects. Do not produce an APIObjectMap directly, instead use [APIObjects](<#APIObjects>) to produce the marshalled yaml representation from CUE data, then provide the result to [Component](<#Component>).
```go
type APIObjectMap map[Kind]map[InternalLabel]string
```
<a name="APIObjects"></a>
## type APIObjects {#APIObjects}
APIObjects represents Kubernetes API objects defined directly from CUE code. Useful to mix in resources to any kind of [Component](<#Component>), for example adding an ExternalSecret resource to a [HelmChart](<#HelmChart>).
[Kind](<#Kind>) must be the resource kind, e.g. Deployment or Service.
[InternalLabel](<#InternalLabel>) is an arbitrary internal identifier to uniquely identify the resource within the context of a \`holos\` command. Holos will never write the intermediate label to rendered output.
Refer to [Component](<#Component>) which accepts an [APIObjectMap](<#APIObjectMap>) field provided by [APIObjects](<#APIObjects>).
```go
type APIObjects struct {
APIObjects map[Kind]map[InternalLabel]APIObject `json:"apiObjects"`
APIObjectMap APIObjectMap `json:"apiObjectMap"`
}
```
<a name="BuildPlan"></a>
## type BuildPlan {#BuildPlan}
BuildPlan represents a build plan for the holos cli to execute. The purpose of a BuildPlan is to define one or more [Component](<#Component>) kinds. For example a [HelmChart](<#HelmChart>), [KustomizeBuild](<#KustomizeBuild>), or [KubernetesObjects](<#KubernetesObjects>).
A BuildPlan usually has an additional empty [KubernetesObjects](<#KubernetesObjects>) for the purpose of using the [Component](<#Component>) DeployFiles field to deploy an ArgoCD or Flux gitops resource for the holos component.
```go
type BuildPlan struct {
Kind string `json:"kind" cue:"\"BuildPlan\""`
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha3\""`
Spec BuildPlanSpec `json:"spec"`
}
```
<a name="BuildPlanComponents"></a>
## type BuildPlanComponents {#BuildPlanComponents}
```go
type BuildPlanComponents struct {
Resources map[InternalLabel]KubernetesObjects `json:"resources,omitempty"`
KubernetesObjectsList []KubernetesObjects `json:"kubernetesObjectsList,omitempty"`
HelmChartList []HelmChart `json:"helmChartList,omitempty"`
KustomizeBuildList []KustomizeBuild `json:"kustomizeBuildList,omitempty"`
}
```
<a name="BuildPlanSpec"></a>
## type BuildPlanSpec {#BuildPlanSpec}
BuildPlanSpec represents the specification of the build plan.
```go
type BuildPlanSpec struct {
// Disabled causes the holos cli to take no action over the [BuildPlan].
Disabled bool `json:"disabled,omitempty"`
// Components represents multiple [HolosComponent] kinds to manage.
Components BuildPlanComponents `json:"components,omitempty"`
}
```
<a name="Chart"></a>
## type Chart {#Chart}
Chart represents a helm chart.
```go
type Chart struct {
// Name represents the chart name.
Name string `json:"name"`
// Version represents the chart version.
Version string `json:"version"`
// Release represents the chart release when executing helm template.
Release string `json:"release"`
// Repository represents the repository to fetch the chart from.
Repository Repository `json:"repository,omitempty"`
}
```
<a name="Component"></a>
## type Component {#Component}
Component defines the fields common to all holos component kinds. Every holos component kind should embed Component.
```go
type Component struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind"`
// APIVersion represents the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"\"v1alpha3\""`
// Metadata represents data about the holos component such as the Name.
Metadata Metadata `json:"metadata"`
// APIObjectMap holds the marshalled representation of api objects. Useful to
// mix in resources to each Component type, for example adding an
// ExternalSecret to a [HelmChart] Component. Refer to [APIObjects].
APIObjectMap APIObjectMap `json:"apiObjectMap,omitempty"`
// DeployFiles represents file paths relative to the cluster deploy directory
// with the value representing the file content. Intended for defining the
// ArgoCD Application resource or Flux Kustomization resource from within CUE,
// but may be used to render any file related to the build plan from CUE.
DeployFiles FileContentMap `json:"deployFiles,omitempty"`
// Kustomize represents a kubectl kustomize build post-processing step.
Kustomize `json:"kustomize,omitempty"`
// Skip causes holos to take no action regarding this component.
Skip bool `json:"skip" cue:"bool | *false"`
}
```
<a name="FileContent"></a>
## type FileContent {#FileContent}
FileContent represents file contents.
```go
type FileContent string
```
<a name="FileContentMap"></a>
## type FileContentMap {#FileContentMap}
FileContentMap represents a mapping of file paths to file contents.
```go
type FileContentMap map[FilePath]FileContent
```
<a name="FilePath"></a>
## type FilePath {#FilePath}
FilePath represents a file path.
```go
type FilePath string
```
<a name="HelmChart"></a>
## type HelmChart {#HelmChart}
HelmChart represents a holos component which wraps around an upstream helm chart. Holos orchestrates helm by providing values obtained from CUE, renders the output using \`helm template\`, then post\-processes the helm output yaml using the general functionality provided by [Component](<#Component>), for example [Kustomize](<#Kustomize>) post\-rendering and mixing in additional kubernetes api objects.
```go
type HelmChart struct {
Component `json:",inline"`
Kind string `json:"kind" cue:"\"HelmChart\""`
// Chart represents a helm chart to manage.
Chart Chart `json:"chart"`
// ValuesContent represents the values.yaml file holos passes to the `helm
// template` command.
ValuesContent string `json:"valuesContent"`
// EnableHooks enables helm hooks when executing the `helm template` command.
EnableHooks bool `json:"enableHooks" cue:"bool | *false"`
}
```
<a name="InternalLabel"></a>
## type InternalLabel {#InternalLabel}
InternalLabel is an arbitrary unique identifier internal to holos itself. The holos cli is expected to never write a InternalLabel value to rendered output files, therefore use a [InternalLabel](<#InternalLabel>) when the identifier must be unique and internal. Defined as a type for clarity and type checking.
A InternalLabel is useful to convert a CUE struct to a list, for example producing a list of [APIObject](<#APIObject>) resources from an [APIObjectMap](<#APIObjectMap>). A CUE struct using InternalLabel keys is guaranteed to not lose data when rendering output because a InternalLabel is expected to never be written to the final output.
```go
type InternalLabel string
```
<a name="Kind"></a>
## type Kind {#Kind}
Kind is a kubernetes api object kind. Defined as a type for clarity and type checking.
```go
type Kind string
```
<a name="KubernetesObjects"></a>
## type KubernetesObjects {#KubernetesObjects}
KubernetesObjects represents a [Component](<#Component>) composed of Kubernetes API objects provided directly from CUE using [APIObjects](<#APIObjects>).
```go
type KubernetesObjects struct {
Component `json:",inline"`
Kind string `json:"kind" cue:"\"KubernetesObjects\""`
}
```
<a name="Kustomize"></a>
## type Kustomize {#Kustomize}
Kustomize represents resources necessary to execute a kustomize build. Intended for at least two use cases:
1. Process a [KustomizeBuild](<#KustomizeBuild>) [Component](<#Component>) which represents raw yaml file resources in a holos component directory.
2. Post process a [HelmChart](<#HelmChart>) [Component](<#Component>) to inject istio, patch jobs, add custom labels, etc...
```go
type Kustomize struct {
// KustomizeFiles holds file contents for kustomize, e.g. patch files.
KustomizeFiles FileContentMap `json:"kustomizeFiles,omitempty"`
// ResourcesFile is the file name used for api objects in kustomization.yaml
ResourcesFile string `json:"resourcesFile,omitempty"`
}
```
<a name="KustomizeBuild"></a>
## type KustomizeBuild {#KustomizeBuild}
KustomizeBuild represents a [Component](<#Component>) that renders plain yaml files in the holos component directory using \`kubectl kustomize build\`.
```go
type KustomizeBuild struct {
Component `json:",inline"`
Kind string `json:"kind" cue:"\"KustomizeBuild\""`
}
```
<a name="Metadata"></a>
## type Metadata {#Metadata}
Metadata represents data about the object such as the Name.
```go
type Metadata struct {
// Name represents the name of the holos component.
Name string `json:"name"`
// Namespace is the primary namespace of the holos component. A holos
// component may manage resources in multiple namespaces, in this case
// consider setting the component namespace to default.
//
// This field is optional because not all resources require a namespace,
// particularly CRDs and DeployFiles functionality.
// +optional
Namespace string `json:"namespace,omitempty"`
}
```
<a name="Platform"></a>
## type Platform {#Platform}
Platform represents a platform to manage. A Platform resource informs holos which components to build. The platform resource also acts as a container for the platform model form values provided by the PlatformService. The primary use case is to collect the cluster names, cluster types, platform model, and holos components to build into one resource.
```go
type Platform struct {
// Kind is a string value representing the resource this object represents.
Kind string `json:"kind" cue:"\"Platform\""`
// APIVersion represents the versioned schema of this representation of an object.
APIVersion string `json:"apiVersion" cue:"string | *\"v1alpha3\""`
// Metadata represents data about the object such as the Name.
Metadata PlatformMetadata `json:"metadata"`
// Spec represents the specification.
Spec PlatformSpec `json:"spec"`
}
```
<a name="PlatformMetadata"></a>
## type PlatformMetadata {#PlatformMetadata}
```go
type PlatformMetadata struct {
// Name represents the Platform name.
Name string `json:"name"`
}
```
<a name="PlatformSpec"></a>
## type PlatformSpec {#PlatformSpec}
PlatformSpec represents the specification of a Platform. Think of a platform specification as a list of platform components to apply to a list of kubernetes clusters combined with the user\-specified Platform Model.
```go
type PlatformSpec struct {
// Model represents the platform model holos gets from from the
// PlatformService.GetPlatform rpc method and provides to CUE using a tag.
Model structpb.Struct `json:"model" cue:"{...}"`
// Components represents a list of holos components to manage.
Components []PlatformSpecComponent `json:"components"`
}
```
<a name="PlatformSpecComponent"></a>
## type PlatformSpecComponent {#PlatformSpecComponent}
PlatformSpecComponent represents a holos component to build or render.
```go
type PlatformSpecComponent struct {
// Path is the path of the component relative to the platform root.
Path string `json:"path"`
// Cluster is the cluster name to provide when rendering the component.
Cluster string `json:"cluster"`
}
```
<a name="Repository"></a>
## type Repository {#Repository}
Repository represents a helm chart repository.
```go
type Repository struct {
Name string `json:"name"`
URL string `json:"url"`
}
```
Generated by [gomarkdoc](<https://github.com/princjef/gomarkdoc>)

View File

@@ -0,0 +1,168 @@
<!-- Code generated by gomarkdoc. DO NOT EDIT -->
# v1alpha3
```go
import "github.com/holos-run/holos/api/schema/v1alpha3"
```
Package v1alpha3 contains CUE definitions intended as convenience wrappers around the core data types defined in package core. The purpose of these wrappers is to make life easier for platform engineers by reducing boiler plate code and generating component build plans in a consistent manner.
## Index
- [type ArgoConfig](<#ArgoConfig>)
- [type Cluster](<#Cluster>)
- [type Fleet](<#Fleet>)
- [type Helm](<#Helm>)
- [type Platform](<#Platform>)
- [type StandardFleets](<#StandardFleets>)
<a name="ArgoConfig"></a>
## type ArgoConfig {#ArgoConfig}
ArgoConfig represents the ArgoCD GitOps configuration for a Component. Useful to define once at the root of the Platform configuration and reuse across all Components.
```go
type ArgoConfig struct {
// Enabled causes holos to render an ArgoCD Application resource for GitOps if true.
Enabled bool `cue:"true | *false"`
// ClusterName represents the cluster within the platform the Application
// resource is intended for.
ClusterName string
// DeployRoot represents the path from the git repository root to the `deploy`
// rendering output directory. Used as a prefix for the
// Application.spec.source.path field.
DeployRoot string `cue:"string | *\".\""`
// RepoURL represents the value passed to the Application.spec.source.repoURL
// field.
RepoURL string
// TargetRevision represents the value passed to the
// Application.spec.source.targetRevision field. Defaults to the branch named
// main.
TargetRevision string `cue:"string | *\"main\""`
}
```
<a name="Cluster"></a>
## type Cluster {#Cluster}
Cluster represents a cluster managed by the Platform.
```go
type Cluster struct {
// Name represents the cluster name, for example "east1", "west1", or
// "management".
Name string `json:"name"`
// Primary represents if the cluster is marked as the primary among a set of
// candidate clusters. Useful for promotion of database leaders.
Primary bool `json:"primary" cue:"true | *false"`
}
```
<a name="Fleet"></a>
## type Fleet {#Fleet}
Fleet represents a named collection of similarly configured Clusters. Useful to segregate workload clusters from their management cluster.
```go
type Fleet struct {
Name string `json:"name"`
// Clusters represents a mapping of Clusters by their name.
Clusters map[string]Cluster `json:"clusters" cue:"{[Name=_]: name: Name}"`
}
```
<a name="Helm"></a>
## type Helm {#Helm}
Helm provides a BuildPlan via the Output field which contains one HelmChart from package core. Useful as a convenience wrapper to render a HelmChart with optional mix\-in resources and Kustomization post\-processing.
```go
type Helm struct {
// Name represents the chart name.
Name string
// Version represents the chart version.
Version string
// Namespace represents the helm namespace option when rendering the chart.
Namespace string
// Resources are kubernetes api objects to mix into the output.
Resources map[string]any `cue:"{...}"`
// Repo represents the chart repository
Repo struct {
Name string `json:"name"`
URL string `json:"url"`
}
// Values represents data to marshal into a values.yaml for helm.
Values interface{} `cue:"{...}"`
// Chart represents the derived HelmChart for inclusion in the BuildPlan
// Output field value. The default HelmChart field values are derived from
// other Helm field values and should be sufficient for most use cases.
Chart core.HelmChart
// EnableKustomizePostProcessor processes helm output with kustomize if true.
EnableKustomizePostProcessor bool `cue:"true | *false"`
// KustomizeFiles represents additional files to include in a Kustomization
// resources list. Useful to patch helm output. The implementation is a
// struct with filename keys and structs as values. Holos encodes the struct
// value to yaml then writes the result to the filename key. Component
// authors may then reference the filename in the kustomization.yaml resources
// or patches lists.
// Requires EnableKustomizePostProcessor: true.
KustomizeFiles map[string]any `cue:"{[string]: {...}}"`
// KustomizePatches represents patches to apply to the helm output. Requires
// EnableKustomizePostProcessor: true.
KustomizePatches map[core.InternalLabel]any `cue:"{[string]: {...}}"`
// KustomizeResources represents additional resources files to include in the
// kustomize resources list.
KustomizeResources map[string]any `cue:"{[string]: {...}}"`
// ArgoConfig represents the ArgoCD GitOps configuration for this Component.
ArgoConfig ArgoConfig
// Output represents the derived BuildPlan for the Holos cli to render.
Output core.BuildPlan
}
```
<a name="Platform"></a>
## type Platform {#Platform}
Platform is a convenience structure to produce a core Platform specification value in the Output field. Useful to collect components at the root of the Platform configuration tree as a struct, which are automatically converted into a list for the core Platform spec output.
```go
type Platform struct {
// Name represents the Platform name.
Name string `cue:"string | *\"holos\""`
// Components is a structured map of components to manage by their name.
Components map[string]core.PlatformSpecComponent
// Model represents the Platform model holos gets from from the
// PlatformService.GetPlatform rpc method and provides to CUE using a tag.
Model structpb.Struct `cue:"{...}"`
// Output represents the core Platform spec for the holos cli to iterate over
// and render each listed Component, injecting the Model.
Output core.Platform
}
```
<a name="StandardFleets"></a>
## type StandardFleets {#StandardFleets}
StandardFleets represents the standard set of Clusters in a Platform segmented into Fleets by their purpose. The management Fleet contains a single Cluster, for example a GKE autopilot cluster with no workloads deployed for reliability and cost efficiency. The workload Fleet contains all other Clusters which contain workloads and sync Secrets from the management cluster.
```go
type StandardFleets struct {
// Workload represents a Fleet of zero or more workload Clusters.
Workload Fleet `json:"workload" cue:"{name: \"workload\"}"`
// Management represents a Fleet with one Cluster named management.
Management Fleet `json:"management" cue:"{name: \"management\", clusters: management: _}"`
}
```
Generated by [gomarkdoc](<https://github.com/princjef/gomarkdoc>)

3
doc/md/backstory.md Normal file
View File

@@ -0,0 +1,3 @@
# Backstory
Holos is a tool intended to lighten the burden of managing Kubernetes resources. In 2020 we set out to develop a holistic platform composed from open source cloud native components. We quickly became frustrated with how each of the major components packaged and distributed their software in a different way. Many projects choose to distribute their software with Helm charts, while others provide plain yaml files and Kustomize bases. The popular Kube Prometheus Stack project provides Jsonnet to render and update Kubernetes yaml manifests.

3
doc/md/cli.md Normal file
View File

@@ -0,0 +1,3 @@
# CLI
Use the `holos` command line interface (CLI) to render individual platform components, entire platforms, and push/pull the platform model.

65
doc/md/comparison.md Normal file
View File

@@ -0,0 +1,65 @@
# Comparison with other tools
:::tip
Holos is designed to complement and improve, not replace, existing tools in the cloud native ecosystem.
:::
## Helm
### Chart Users
Describe how things are different when using an upstream helm chart.
### Chart Authors
Describe how things are different when writing a new helm chart.
## Kustomize
TODO
## ArgoCD
TODO
## Flux
TODO
## Timoni
| Aspect | Timoni | Holos | Comment |
| -- | -- | -- | -- |
| Language | CUE | CUE | Like Holos, Timoni is also built on CUE. |
| Artifact | OCI Image | Plain YAML Files | The Holos Authors find plain files easier to work with and reason about than OCI images. |
| Outputs to | OCI Image Repository | Local Git repository | Holos is designed for use with existing GitOps tools. |
| Concept | Module | Component | A Timoni Module is analogous to a Holos Component. |
| Concept | Bundle | Platform | A Timoni Bundle is somewhat similar, but smaller in scope to a Holos Platform. |
:::important
The Holos Authors are deeply grateful to Stefan and Timoni for the capability of
importing Kubernetes custom resource definitions into CUE. Without this
functionality, much of the Kubernetes ecosystem would be more difficult to
manage in CUE and therefore in Holos.
:::
## KubeVela
1. Also built on CUE.
2. Intended to create an Application abstraction.
3. Holos prioritizes composition over abstraction.
4. An abstraction of an Application acts as a filter that removes all but the lowest common denominator functionality. The Holos Authors have found this filtering effect to create excessive friction for software developers.
5. Holos focuses instead on composition to empower developers and platform engineers to leverage the unique features and functionality of their software and platform.
## Pulumi
TODO
## Jsonnet
TODO

341
doc/md/concepts.md Normal file
View File

@@ -0,0 +1,341 @@
# Concepts
## Introduction
This page is intended as a high level conceptual overview of the key concepts in
Holos. Refer to the [Core API](/docs/api/core/) for low level reference
documentation.
Holos is a tool built for platform engineers. The Holos authors share three
core values which guide our design decisions for the tool.
1. Safety
2. Ease of use
3. Consistency
Each of the following concepts are intended to support and strengthen one or
more of these core values. In this way we hope to lighten the burden carried by
platform engineers.
## Concepts
- [Component](<#component>) - The primary building block in Holos, wraps a Helm chart, Kustomize base, or plain resources defined in CUE.
- [Platform](<#platform>) - A collection of Components integrated into a software development platform.
- [Model](<#model>) - Structured data included in the Platform specification, available to all Components. For example, your organization's domain name.
- [Rendering](<#rendering>) - Holos is a tool that makes the process of rendering Kubernetes manifests safer, easier, and consistent.
```mermaid
graph BT
Platform[<a href="#platform">Platform</a>]
Component[<a href="#component">Components</a>]
Helm[<a href="#component">Helm</a>]
Kustomize[<a href="#component">Kustomize</a>]
CUE[<a href="#component">CUE</a>]
Component --> Platform
Helm --> Component
Kustomize --> Component
CUE --> Component
```
<!--
```mermaid
---
title: Figure 1 - Holos Concepts
---
mindmap
root((Holos))
Platform
Components
HelmChart
KustomizeBuild
KubernetesObjects
Model
name: Example Org
domain: example.com
Renders
YAML Files
Kubernetes Manifests
ArgoCD Application
FluxCD Kustomization
```
-->
## Component
A Component is the primary building block when managing software with Holos. A
software project you wish to integrate into your platform, for example ArgoCD,
is managed using one or more components.
The primary Component kinds are:
1. **HelmChart** to render config provided by Helm.
2. **KustomizeBuild** to render config provided by Kustomize.
3. **KubernetesObjects** to render config provided by CUE.
Components are intended to integrate unmodified upstream software releases into
your Platform. In this way, the focus of a Component is more about the unique
differentiating aspects of your platform than the upstream software contained in
the Component.
#### Example HelmChart Component
The ArgoCD Component is a good example of a HelmChart component because it takes
advantage of most of the key features that empower you to focus on the key
differentiators of your unique platform.
Take note of the following key points in this example ArgoCD Component:
1. The Component wraps the ArgoCD Helm Chart in a way that's easy to upgrade and maintain over time.
2. Newer Gateway API resources are mixed-in replacing the older Ingress resource included in the chart.
3. Helm output is passed through Kustomize to configure secure mutual TLS encryption.
4. Helm values are easier and safer to manipulate with CUE instead of text markup.
5. Kustomize is easier and safer to manipulate with CUE instead of text markup.
6. Platform data Model values are easily accessible, for example the OIDC issuer and the organizations's domain name.
The Component wraps around the unmodified upstream ArgoCD helm chart
providing easier upgrades as new versions are released.
Note how the Component facilitates composition by allowing us to mix-in new
functionality from the ecosystem without modifying the upstream chart. The
Platform this Component integrates with uses the new Gateway API, but the
upstream helm chart does not yet support Gateway API. See how the Resources
field is used to mix-in a ReferenceGrant from the Gateway API without modifying
the upstream helm chart.
The Platform uses Istio to implement service to service encryption with mutual
TLS. The Component passes the Helm output to Kustomize to integrate with Istio.
Kustomize is used to patch the argocd-server Deployment resource to inject the
Istio sidecar for mutual TLS.
Helm values are safer and easier to work with in CUE. Note how you can modify
helm values using well defined data instead of manipulating text yaml files.
Similarly, the yaml files used for Kustomize are produced by CUE, which is again
safer and easier because the Kustomize spec has been imported into CUE and is
validated.
Finally, the domain name used by this Platform is easily accessible from the
PlatformSpec which is defined at the root level and made available to all
components integrated into the platform. Similarly, data values shared by all
of the Components that make up ArgoCD is defined in a structure accessible by
each of these components.
```cue
package holos
import (
"encoding/yaml"
"strings"
)
// Produce a helm chart build plan.
(#Helm & Chart).Output
let Chart = {
Name: "argo-cd"
Namespace: "argocd"
Version: "7.1.1"
Chart: chart: release: "argocd"
// The upstream chart uses a Job to create the argocd-redis Secret. Enable
// hooks to enable the Job.
Chart: enableHooks: true
Repo: name: "argocd"
Repo: url: "https://argoproj.github.io/argo-helm"
// Ensure all of our mix-in resources go into the same namespace as the Chart.
Resources: [_]: [_]: metadata: namespace: Namespace
// Grant the Gateway namespace the ability to refer to the backend service
// from HTTPRoute resources.
Resources: ReferenceGrant: (#IstioGatewaysNamespace): #ReferenceGrant
// Pass the helm output through kustomize.
EnableKustomizePostProcessor: true
// Force all resources into the component namespace, some resources in the
// helm chart do not specify the namespace so they will get mis-applied
// when the kubectl (client-go) context is another namespace.
KustomizeFiles: "kustomization.yaml": namespace: Namespace
// Patch the backend with the service mesh sidecar.
KustomizePatches: {
mesh: {
target: {
group: "apps"
version: "v1"
kind: "Deployment"
name: "argocd-server"
}
patch: yaml.Marshal(IstioInject)
}
}
Values: #Values & {
kubeVersionOverride: "1.29.0"
// handled in the argo-crds component
crds: install: false
global: domain: "argocd.\(_Platform.Model.org.domain)"
dex: enabled: false
// the service mesh handles secure mTLS
configs: params: "server.insecure": true
configs: cm: {
"admin.enabled": false
"oidc.config": yaml.Marshal(OIDCConfig)
"users.anonymous.enabled": "false"
}
// Refer to https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/
let Policy = [
"g, argocd-view, role:readonly",
"g, prod-cluster-view, role:readonly",
"g, prod-cluster-edit, role:readonly",
"g, prod-cluster-admin, role:admin",
]
configs: rbac: "policy.csv": strings.Join(Policy, "\n")
}
}
let IstioInject = [{
op: "add",
path: "/spec/template/metadata/labels/sidecar.istio.io~1inject",
value: "true",
}]
let OIDCConfig = {
name: "Holos Platform"
issuer: _ArgoCD.issuerURL
clientID: _ArgoCD.clientID
requestedScopes: _ArgoCD.scopesList
// Set redirect uri to https://argocd.example.com/pkce/verify
enablePKCEAuthentication: true
// groups is essential for rbac
requestedIDTokenClaims: groups: essential: true
}
```
## Platform
A Platform refers to all of the software and services integrated together to
provide your organization's software development platform. Holos is designed to
manage all of the resources that compose your Platform using the [Kubernetes
Resource Model][krm] (KRM). Nearly all platforms are larger than Kubernetes
itself. For example, your developers likely need a GCS or S3 bucket to store
data. Holos takes advantage of Crossplane to manage resources in a consistent
way.
Holos defines a [Platform][Platform] object which collects multiple Components
together along with organizational data defined by your Model. Consider the
following example, which is a Platform that manages a single Component which
manages namespaces for each cluster in the Platform.
```cue
package holos
import v1 "github.com/holos-run/holos/api/v1alpha2"
v1.#Platform & {
metadata: name: "example"
spec: components: [{
path: "components/namespaces"
cluster: "cluster1"
}]
}
```
This platform is rendered by the command:
```bash
holos render platform ./platform
```
When Holos renders the platform, it iterates over each component, generates and
executes a [BuildPlan][BuildPlan], then writes the fully rendered output of the
component to the filesystem. In this simple example, two files are produced:
1. `deploy/clusters/cluster1/components/namespaces/namespaces.gen.yaml`
2. `deploy/clusters/cluster1/gitops/namespaces.application.gen.yaml`
The first file is a plain kubernetes manifest containing Namespace resources.
The second file is an ArgoCD Application resource to deploy and manage the
resources defined in the first file.
## Model
The Platform Model is where you store top-level data values used throughout
multiple components in your Platform. Your organization's domain name is a
prime example of the kind of data stored in the Model. Many components derive
host names from your organization's domain name. CUE makes this process safe,
easy, and consistent. For example:
```cue
hostname: "argocd.\(_Platform.Model.org.domain)"
```
When Holos renders a Platform, the model is loaded from a JSON file in the local
filesystem. The platform model file is intended to be committed to version
control along with the rest of the Holos Platform and Component code.
Holos additionally provides a web ui and form to make it easy to enter and
validate top level configuration data. You have complete control over the web
form, it's rendered from JSON data defined by CUE. Customizing the web form is
an advanced topic, the key concept to take away is the Model is for top level,
platform-wide data. You control the shape and structure of the Model, and you
have the ability to collect Model values using a simple web form.
## Rendering
Holos uses the Kubernetes resource model to manage configuration. The Holos
command line interface is the primary method you'll use to manage your platform.
Holos uses CUE to provide a unified configuration model of the platform. This
unified configuration is built up from components packaged with Helm, Kustomize,
CUE, or any other tool that can produce Kubernetes resource manifests as output.
This process can be thought of as a data **rendering pipeline**. The key
concept is that Holos will always produce fully rendered output, but delegates
the _application_ of the configuration to other tools like kubectl apply,
ArgoCD, or Flux.
```mermaid
---
title: Figure 2 - Render Pipeline
---
graph LR
PS[<a href="/docs/api/core/v1alpha2#PlatformSpec">PlatformSpec</a>]
BP[<a href="/docs/api/core/v1alpha2#BuildPlan">BuildPlan</a>]
HC[<a href="/docs/api/core/v1alpha2#HolosComponent">Components</a>]
H[<a href="/docs/api/core/v1alpha2#HelmChart">HelmChart</a>]
K[<a href="/docs/api/core/v1alpha2#KustomizeBuild">KustomizeBuild</a>]
O[<a href="/docs/api/core/v1alpha2#KubernetesObjects">KubernetesObjects</a>]
P[<a href="/docs/api/core/v1alpha2#Kustomize">Kustomize</a>]
Y[Kubernetes <br/>Resources]
G[GitOps <br/>Resource]
FS[Local Files]
C[Kube API Server]
PS --> BP --> HC
HC --> H --> P
HC --> K --> P
HC --> O --> P
P --> Y --> FS
P --> G --> FS
FS --> ArgoCD --> C
FS --> Flux --> C
FS --> kubectl --> C
```
[krm]: https://docs.google.com/document/d/1RmHXdLhNbyOWPW_AtnnowaRfGejw-qlKQIuLKQWlwzs/view#heading=h.sa6p0aye4ide
[Platform]: /docs/api/core/v1alpha2/#Platform
[BuildPlan]: /docs/api/core/v1alpha2/#BuildPlan

View File

@@ -0,0 +1,47 @@
# Rendering
:::tip
This document provides a brief overview of the rendering process, a core design
element in Holos.
:::
Holos uses the Kubernetes resource model to manage configuration. The `holos`
command line interface is the primary method you'll use to manage your platform.
Holos uses CUE to provide a unified configuration model of the platform. This
unified configuration is built up from components packaged with Helm, Kustomize,
CUE, or any other tool that can produce Kubernetes resource manifests as output.
This process can be thought of as a data **rendering pipeline**. The key
concept is that `holos` will always produce fully rendered output, but delegates
the _application_ of the configuration to other tools like `kubectl apply`,
ArgoCD, or Flux.
```mermaid
---
title: Figure 2 - Render Pipeline
---
graph LR
PS[<a href="/docs/api/core/v1alpha2#PlatformSpec">PlatformSpec</a>]
BP[<a href="/docs/api/core/v1alpha2#BuildPlan">BuildPlan</a>]
HC[<a href="/docs/api/core/v1alpha2#HolosComponent">HolosComponent</a>]
H[<a href="/docs/api/core/v1alpha2#HelmChart">HelmChart</a>]
K[<a href="/docs/api/core/v1alpha2#KustomizeBuild">KustomizeBuild</a>]
O[<a href="/docs/api/core/v1alpha2#KubernetesObjects">KubernetesObjects</a>]
P[<a href="/docs/api/core/v1alpha2#Kustomize">Kustomize</a>]
Y[Kubernetes <br>Resources]
G[GitOps <br>Resource]
C[Kube API Server]
PS --> BP --> HC
HC --> H --> P
HC --> K --> P
HC --> O --> P
P --> Y --> C
P --> G --> C
```

88
doc/md/glossary.md Normal file
View File

@@ -0,0 +1,88 @@
# Glossary
This page describes the terms used within the context of Holos.
## Platform
In Holos, a Platform is a comprehensive environment configured using the
Kubernetes resource model. It extends beyond traditional Kubernetes
functionality by integrating cloud resources through Crossplane, allowing for a
unified management approach across both Kubernetes and cloud infrastructure. A
Platform typically consists of one Management Cluster, which handles control and
secret management, and one or more Workload Clusters, where application
workloads are deployed and run. This architecture enables a consistent and
scalable approach to managing diverse resources and services within the
cloud-native ecosystem.
## Management Cluster
In the context of Holos, a Management Cluster is a special Kubernetes cluster
that hosts Kubernetes controllers. For example, cert-manager, Cluster api, and
Crossplane. A management cluster manages a single platform. The primary
function of this cluster is to securely store and manage secrets, ensuring the
secure handling of sensitive information such as credentials, API keys, and
other confidential data. The Management Cluster serves as a centralized and
secure control plane for the platform, facilitating the orchestration and
management of other components.
## Workload Cluster
In Holos, a Workload Cluster is a Kubernetes cluster designed to host and run
application workloads. Unlike the Management Cluster, which focuses on control
and secret management, Workload Clusters are dedicated to executing the actual
applications and services. These clusters can vary in size and configuration
based on the specific needs of the applications they support. Workload Clusters
leverage Kubernetes' orchestration capabilities to manage the deployment,
scaling, and operation of containerized applications, providing a flexible and
scalable environment for running production workloads within the platform.
## Platform Form
In Holos, a Platform Form is a customizable web form defined by JSON data. Each
platform within Holos has a unique Platform Form, which serves as an interface
for configuring and managing the platform's settings and resources. Platform
engineers can customize the Platform Form by modifying the underlying CUE
(Configuration Unified Engine) code, allowing for tailored configurations that
meet specific requirements. This flexibility enables platform engineers to
create a user-friendly and specific interface for managing the platform's
components and operations.
## Platform Model
In Holos, the Platform Model represents the collection of values submitted
through the Platform Form. It encapsulates the specific configuration details
and settings defined by the platform engineers, serving as the blueprint for the
platform's setup and operation. The Platform Model is essential for translating
the customized options and parameters from the Platform Form into actionable
configurations within the Holos ecosystem, ensuring that the platform operates
according to the specified requirements and guidelines.
## Secret Store
In Holos, a SecretStore is a repository for securely storing and managing
sensitive data such as passwords, API keys, and other confidential information.
It is compatible with any secret store supported by the External Secrets
Operator. By default, the management cluster serves as the SecretStore to
minimize dependencies and simplify the architecture. This setup ensures that
secrets are managed in a secure and centralized manner, aligning with the
overall security framework of the platform.
## Service Mesh
In Holos, a Service Mesh is a dedicated infrastructure layer for managing,
observing, and securing service-to-service communications within a microservices
architecture. It typically includes features such as load balancing, traffic
routing, service discovery, and security policies like mutual TLS and access
control. The Service Mesh abstracts these functionalities away from the
application code, providing a centralized control plane for managing the
interactions between microservices. This facilitates better observability,
resilience, and security in complex, distributed environments.
## Zero Trust
In the context of Holos and broader security practices, Zero Trust is a security
model that assumes no implicit trust is granted to any user, system, or
component inside or outside the network. Instead, every request for access is
treated as potentially malicious, and verification is required at every stage.
This model enforces strict identity verification, continuous monitoring, and
least-privilege access policies.

Binary file not shown.

After

Width:  |  Height:  |  Size: 934 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 703 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1014 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 728 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1014 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 854 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 MiB

View File

@@ -0,0 +1,6 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# ArgoCD
Coming soon.

View File

@@ -0,0 +1,3 @@
# Backstage
Coming soon.

View File

@@ -0,0 +1,3 @@
# Observability
Coming soon.

17
doc/md/guides/overview.md Normal file
View File

@@ -0,0 +1,17 @@
# Overview
<!-- https://kubernetes.io/docs/contribute/style/diagram-guide/ -->
This tutorial covers the following process of getting started with Holos.
```mermaid
graph LR
A[1. Install <br>holos] -->
B[2. Register <br>account] -->
C[3. Generate <br>platform] -->
D[4. Render <br>platform] -->
E[5. Apply <br>config]
classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000;
class A,B,C,D,E box
```

62
doc/md/guides/register.md Normal file
View File

@@ -0,0 +1,62 @@
# Registration
Holos leverages a simple web app to collect and store platform attributes with a web form. Register an account with the web app to create and retrieve the platform model.
```
holos register user
```
:::tip
Holos allows you to customize all of the sections and fields of your platform model.
:::
## Generate your Platform
Generate your platform configuration from the holos reference platform embedded in the `holos` executable. Platform configuration is stored in a git repository.
```bash
mkdir holos-infra
cd holos-infra
holos generate platform holos
```
The generate command writes many files organized by platform component into the current directory
TODO: Put a table here describing key elements?
:::tip
Take a peek at `holos generate platform --help` to see other platforms embedded in the holos executable.
:::
## Push the Platform Form
```
holos push platform form .
```
## Fill in the form
TODO
## Pull the Platform Model
Once the platform model is saved, pull it into the holos-infra repository:
```
holos pull platform model .
```
## Render the Platform
With the platform model and the platform spec, you're ready to render the complete platform configuration:
```
holos render platform ./platform
```
## Summary

Binary file not shown.

After

Width:  |  Height:  |  Size: 624 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 116 KiB

View File

@@ -0,0 +1,687 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Admonition from '@theme/Admonition';
# Try Holos Locally
This guide walks through the process of building and managing a software
development platform with Holos. The k3d platform built in this guide is a
slimmed down version of the larger, more holistic, Holos reference platform.
Holos is different from existing tools in a few important ways.
1. Holos provides a **unified configuration model** purpose built to improve on
unmodified Helm charts, Kustomize bases, or anything else that produces
structured configuration data.
2. Holos all but **eliminates the need to template yaml**, a common source of
frustration and errors in production.
3. Holos platforms are **composable** and have breadth. The toolchain and
techniques scale down to one machine and up to multiple clusters across
multiple regions.
4. The unified configuration model is well suited to a **Zero Trust security
model**. Platform wide policy configuration is easier to manage with Holos.
---
This guide assumes commands are run locally. Capitalized terms have specific
definitions described in the [Glossary](/docs/glossary).
## What you'll need {#Requirements}
You'll need the following tools installed to complete this guide.
1. [holos](/docs/install) - to build the platform.
2. [helm](https://helm.sh/docs/intro/install/) - to render Holos components that wrap upstream Helm charts.
3. [k3d](https://k3d.io/#installation) - to provide a k8s api server.
4. [OrbStack](https://docs.orbstack.dev/install) or [Docker](https://docs.docker.com/get-docker/) - to use k3d.
5. [kubectl](https://kubernetes.io/docs/tasks/tools/) - to interact with the k8s api server.
6. [mkcert](https://github.com/FiloSottile/mkcert?tab=readme-ov-file#installation) - to make trusted TLS certificates.
7. [jq](https://jqlang.github.io/jq/download/) - to fiddle with JSON output.
:::note
Registering an account **is recommended** to try out proper authentication and
authorization in Holos, but you can complete this guide without signing up.
:::
## Goal {#Goal}
By the end of this guide you'll have built the foundation of a software
development platform. The foundation provides Zero Trust security by
holistically integrating off-the-shelf open source software.
1. Istio is configured to authenticate and authorize requests using an OIDC
ID-Token issued by ZITADEL before requests reach backend services.
2. The platform provides single sign-on and role based access control for all
services running on the platform.
This guide strives to keep things neat and tidy. All of the resources are
located in one k3d cluster and one local Git repository. If you want to clean
up at any point, do so with:
```bash
k3d cluster delete workload
rm -rf holos-k3d
```
## Sign In or Out {#Sign-In}
Holos provides integrated authentication and authorization which we'll use in
this guide to protect a service. We recommend registering an account to see
this in action. Registration also enables you to explore the customizable web
form that simplifies complex configuration.
If you opt-out, the platform will be configured to use a fake identity in place
of real id tokens.
<Tabs groupId="registration">
<TabItem value="registered" label="Sign In">
```bash
holos register user
```
</TabItem>
<TabItem value="unregistered" label="Opt Out">
```bash
holos logout
```
</TabItem>
</Tabs>
## Create the Platform {#Create-Platform}
A server-side platform resource in Holos stores the web form used to simplify
platform wide configuration.
First, initialize an empty Git repository:
```bash
mkdir holos-k3d
cd holos-k3d
git init
```
<Tabs groupId="registration">
<TabItem value="registered" label="Signed In">
Use `holos` to make the rpc call to create the server-side platform
resource.
```bash
holos create platform --name k3d --display-name "Try Holos Locally"
```
</TabItem>
<TabItem value="unregistered" label="Signed Out">
Create a blank `platform.metadata.json` file so subsequent holos commands
skip rpc calls.
```bash
touch platform.metadata.json
```
</TabItem>
</Tabs>
### Generate the Platform {#Generate-Platform}
Generate the platform code in the repository root.
```bash
holos generate platform k3d
```
Commit the generated platform config to the repository.
```bash
git add .
git commit -m "holos generate platform k3d - $(holos --version)"
```
### Push the Platform Form
Each Holos platform has a Platform Form used to submit top level, platform-wide
configuration values. The purpose of the form is to validate configuration
values and simplify complicated configurations and integrations.
<Tabs groupId="registration">
<TabItem value="registered" label="Signed In">
Push the Platform Form to publish it. Browse to the printed URL to view the
form.
```bash
holos push platform form .
```
</TabItem>
<TabItem value="unregistered" label="Signed Out">
You will update the Platform Model locally in a later step so there's
nothing to do in this step. Only signed-in users can push a Platform Form
to the Holos web server.
```bash
# holos push platform form .
```
</TabItem>
</Tabs>
The Platform Form is defined locally in `forms/platform/platform-form.cue`.
On the web it looks like:
![Platform Form Default Values](./form-pushed.png)
### Update the Platform Model {#Platform-Model}
Holos needs initial, top level configuration values to render the platform. The
Platform Model is the term we use for these values. In this section you will
configure role based access control by way of updating the Platform Model.
In the k3d platform you're building now, role based access control is
implemented by asserting against the oidc id token subject. Update the form
with the `sub` claim value from your id token. This will ensure only you have
access to platform services.
<Tabs groupId="registration">
<TabItem value="registered" label="Signed In">
Copy and paste the `sub` value into your Platform Form's Subject field.
```bash
holos login --print-claims --log-level=error | jq -r .sub
```
After pasting the `sub` value, click Submit on the form.
</TabItem>
<TabItem value="unregistered" label="Signed Out">
You don't have an id token when you're signed out, so there's nothing for
you to do in this step.
```bash
# holos login --print-claims --log-level=error | jq -r .sub
```
The platform will be configured to assert against the User-Agent header
instead.
</TabItem>
</Tabs>
### Pull the Platform Model {#Pull-the-Platform-Model}
The Platform Model needs to be pulled into the local Git repository after the
form has been submitted. Next, we'll run `holos render` which operates
exclusively on local files.
Holos stores the Platform Model in the `platform.config.json` file. Holos
provides this file as input to CUE when rendering the platform. This file is
intended to be added to version control.
<Tabs groupId="registration">
<TabItem value="registered" label="Signed In">
Pull the updated Platform Model into the local repository.
```bash
holos pull platform model .
git add platform.config.json
git commit -m "Add platform model"
```
</TabItem>
<TabItem value="unregistered" label="Signed Out">
The holos generate platform k3d command created an initial Platform Model in
`platform.config.json`. As a result there's nothing to do in this step.
```bash
# holos pull platform model .
# git add platform.config.json
# git commit -m "Add platform model"
```
</TabItem>
</Tabs>
## Render the Platform {#Render-the-Platform}
Holos has everything necessary to render the platform once the
`platform.config.json` file and the code from `holos generate` are in the
current directory.
Rendering a platform is the process of iterating over each platform component
and rendering it into plain yaml. Holos does not apply the resulting manifests.
Other tools like kubectl, ArgoCD, or Flux are responsible for applying the
manifests.
```bash
holos render platform ./platform
```
The render command writes the manifest files to the `deploy/` directory. Commit
the files so they can be applied via GitOps later.
```bash
git add deploy
git commit -m "holos render platform ./platform"
```
:::info[Don't blink, this is where Holos builds the platform]
It usually takes no more than a few seconds.
Rendering the holos reference platform currently results in about 500K lines of
yaml. In contrast, roughly 80K lines are produced by this slimmed down k3d
platform.
We mention this because the scale doesn't matter as much as it does with other
tools. Manage millions of lines of configuration with Holos the same way this
guide manages thousands. This is made possible by the unique way CUE unifies
all configuration into one single model.
:::
## Configure DNS {#DNS}
Configure your machine to resolve `*.holos.localhost` to your loopback
interface. This is necessary for requests to reach the workload cluster.
<Tabs>
<TabItem value="macos" label="macOS" default>
Cache sudo credentials.
Admin access is necessary to setup a local dnsmasq instance and configure
macOS's DNS resolver.
```bash
sudo -v
```
Resolve *.holos.localhost DNS queries to 127.0.0.1.
```bash
bash ./scripts/local-dns
```
</TabItem>
<TabItem value="linux" label="Linux">
[NSS-myhostname](http://man7.org/linux/man-pages/man8/nss-myhostname.8.html)
ships with many Linux distributions and should resolve *.localhost
automatically to 127.0.0.1.
Otherwise it is installable with:
```bash
sudo apt install libnss-myhostname
```
</TabItem>
<TabItem value="windows" label="Windows">
Ensure the loopback interface has at least the following names in `C:\windows\system32\drivers\etc\hosts`
```
127.0.0.1 httpbin.holos.localhost app.holos.localhost
```
</TabItem>
</Tabs>
## Create the Cluster {#Create-Cluster}
The Workload Cluster is where your applications and services will be deployed.
In production this is usually an EKS, GKE, or AKS cluster.
:::tip
Holos supports all compliant Kubernetes clusters. Holos was developed and tested
on GKE, EKS, Talos, k3s, and Kubeadm clusters.
:::
<Tabs>
<TabItem value="evaluate" label="Try Holos" default>
Use this command when exploring Holos.
```bash
k3d cluster create workload \
--port "443:443@loadbalancer" \
--k3s-arg "--disable=traefik@server:0"
```
</TabItem>
<TabItem value="develop" label="Develop Holos">
Use this command when developing Holos.
```bash
k3d registry create registry.holos.localhost --port 5100
```
```bash
k3d cluster create workload \
--registry-use k3d-registry.holos.localhost:5100 \
--port "443:443@loadbalancer" \
--k3s-arg "--disable=traefik@server:0"
```
</TabItem>
</Tabs>
Traefik is disabled because Istio provides the same functionality.
## Apply the Platform Components {#Apply-Platform-Components}
Use `kubectl` to apply each platform component. In production, it's common to
fully automate this process with ArgoCD, but we use `kubectl` to the same
effect.
### Local CA {#Local-CA}
Holos platforms use cert manager to issue tls certificates. The browser and
tools we're using need to trust these certificates to work together.
Admin access is necessary for `mkcert` to manage the certificate into your trust
stores.
```bash
sudo -v
```
Manage the local CA and copy the CA key to the workload cluster so that cert
manager can manage trusted certificates.
```bash
bash ./scripts/local-ca
```
:::warning
Take care to run the local-ca script each time you create the workload cluster
so that Certificates are issued correctly.
:::
### Service Mesh
The platform service mesh provides an ingress gateway and connectivity useful
for observability, reliability, and security.
#### Namespaces
With Holos, components are automatically added to the namespaces component,
useful for centrally managed policies.
```bash
kubectl apply --server-side=true -f ./deploy/clusters/workload/components/namespaces
```
#### Custom Resource Definitions
```bash
kubectl apply --server-side=true -f ./deploy/clusters/workload/components/gateway-api
kubectl apply --server-side=true -f ./deploy/clusters/workload/components/istio-base
```
#### Cert Manager {#cert-manager}
Apply the cert-manager controller.
```bash
kubectl apply --server-side=true -f ./deploy/clusters/workload/components/cert-manager
```
Apply the ClusterIssuer which issues Certificate resources using the local
certificate authority.
```bash
kubectl -n cert-manager wait pod -l app.kubernetes.io/component=webhook --for=condition=Ready
kubectl apply --server-side=true -f deploy/clusters/workload/components/local-ca
kubectl apply --server-side=true -f deploy/clusters/workload/components/certificates
kubectl -n istio-gateways wait certificate httpbin.holos.localhost --for=condition=Ready
```
:::warning
The certificate will time out before becoming ready if the [local-ca](#Local-CA)
script was not run after the cluster was created.
:::
#### Istio {#Istio}
Istio implements the Service Mesh.
```bash
kubectl apply --server-side=true -f ./deploy/clusters/workload/components/istio-cni
kubectl apply --server-side=true -f ./deploy/clusters/workload/components/istiod
kubectl apply --server-side=true -f ./deploy/clusters/workload/components/gateway
```
Verify the Gateway is programmed and the listeners have been accepted:
```bash
kubectl -n istio-gateways wait gateway default --for=condition=Accepted
```
#### httpbin {#httpbin}
httpbin is a simple backend service useful for end-to-end testing.
```bash
kubectl apply --server-side=true -f deploy/clusters/workload/components/httpbin-backend
kubectl apply --server-side=true -f deploy/clusters/workload/components/httpbin-routes
kubectl -n holos-system wait pod -l app.kubernetes.io/instance=httpbin --for=condition=Ready
```
:::info
Browse to [https://httpbin.holos.localhost/](https://httpbin.holos.localhost/)
to verify end to end connectivity. You should see the httpbin index page.
:::
### Authenticating Proxy
The auth proxy is responsible for authenticating browser requests, handling the
oidc authentication flow, and providing a signed id token to the rest of the
services in the mesh.
#### Cookie Secret
The auth proxy stores session information in an encrypted cookie. Generate a
random cookie encryption Secret and apply.
```bash
LC_ALL=C tr -dc A-Za-z0-9 </dev/urandom \
| head -c 32 \
| kubectl create secret generic "authproxy" \
--from-file=cookiesecret=/dev/stdin \
--dry-run=client -o yaml \
| kubectl apply -n istio-gateways -f-
```
#### Deployment
The auth proxy Deployment receives requests from web browsers and responds with
an authentication decision.
```bash
kubectl apply --server-side=true -f deploy/clusters/workload/components/authproxy
kubectl apply --server-side=true -f deploy/clusters/workload/components/authroutes
```
<Tabs groupId="registration">
<TabItem value="registered" label="Signed In">
<Admonition type="info">
Verify authentication is working by browsing to
[https://httpbin.holos.localhost/holos/authproxy](https://httpbin.holos.localhost/holos/authproxy).
We want a simple `Authenticated` response.
<Admonition type="tip">
You may need to refresh the page a few times while the platform configures
itself.
</Admonition>
</Admonition>
Istio will respond with `no healthy upstream` until the pod becomes ready.
Wait for the pod to become ready with:
```bash
kubectl -n holos-system wait pod -l app.kubernetes.io/instance=httpbin --for=condition=Ready
```
Once authenticated, visit
[https://httpbin.holos.localhost/holos/authproxy/userinfo](https://httpbin.holos.localhost/holos/authproxy/userinfo)
which returns a subset of claims from your id token.
<Admonition type="warning">
If you get `Unauthorized` instead of a json response body, make sure you
[authenticated](https://httpbin.holos.localhost/holos/authproxy) first.
</Admonition>
```json
{
"user": "275552236589843464",
"email": "demo@holos.run",
"preferredUsername": "demo"
}
```
</TabItem>
<TabItem value="unregistered" label="Signed Out">
The auth proxy will always try to sign you in when you are signed out, so
there isn't much to do here. Please do take a moment to glance at the
Signed In tab to see how this would work if you were signed in.
The `k3d` platform relies on `https://login.holos.run` to issue id tokens.
Authorization has been configured against fake request headers instead of
the real `x-oidc-id-token` header.
</TabItem>
</Tabs>
### Authorization Policy
Configure authorization policies using attributes of the authenticated request.
Authorization policies route web requests through the auth proxy and then
validate all requests against the `x-oidc-id-token` header.
```bash
kubectl apply --server-side=true -f deploy/clusters/workload/components/authpolicy
```
Istio make take a few seconds to program the Gateway with the
AuthorizationPolicy resources.
## Try out Zero Trust
A basic Zero Trust security model is now in place. The platform authenticates
and authorizes requests before they reach the backend service.
### Browser
<Tabs groupId="registration">
<TabItem value="registered" label="Signed In">
The platform has been configured to authorize requests with a `x-oidc-id-token` header.
1. Verify authentication is working by browsing to [https://httpbin.holos.localhost/dump/request](https://httpbin.holos.localhost/dump/request).
- Refresh the page a few times.
- The `httpbin` backend pods should echo back the `x-oidc-id-token`
header injected by the auth proxy.
2. Note the `x-oidc-id-token` header is not sent by your browser but is
received by the backend service.
- This design reduces the risk of exposing id tokens in the browser.
- Browser request size remains constant as more claims are added to id
tokens.
- Reliability improves because id tokens often overflow request header
buffers when they pass through middle boxes across the internet.
</TabItem>
<TabItem value="unregistered" label="Signed Out">
The platform has been configured to authorize requests with a `User-Agent: anonymous` header.
1. Open an incognito window (Cmd+Shift+N) to verify the platform is
enforcing the authorization policy.
2. Browse to
[https://httpbin.holos.localhost/dump/request](https://httpbin.holos.localhost/dump/request)
you should be redirected to the sign in page by the auth proxy.
- You **do not** need to register or sign in.
- This step verifies the platform is redirecting unauthenticated
requests to the identity provider.
- Navigate back or close and re-open an incognito window.
3. Set your `User-Agent` header to `anonymous` using your browser developer tools.
- For Chrome the process is described
[here](https://developer.chrome.com/docs/devtools/device-mode/override-user-agent#override_the_user_agent_string).
- The purpose is to simulate an authenticated request.
4. Browse to
[https://httpbin.holos.localhost/dump/request](https://httpbin.holos.localhost/dump/request).
- The platform should allow the request through to the backend pod.
- `httpbin` should echo back your request which should contain `User-Agent: anonymous`.
</TabItem>
</Tabs>
### Command Line
Verify unauthenticated requests are blocked by default outside the browser.
```bash
curl -I https://httpbin.holos.localhost/dump/request
```
You should receive a `HTTP/2 302` response that redirects to `location:
https://login.holos.run` to start the oauth login flow.
Next, verify authenticated requests are allowed.
<Tabs groupId="registration">
<TabItem value="registered" label="Signed In">
The platform is configured to authenticate the id token present in the
`x-oidc-id-token` header.
💡 It also works with `grpcurl`.
```bash
curl -H x-oidc-id-token:$(holos token) https://httpbin.holos.localhost/dump/request
```
</TabItem>
<TabItem value="unregistered" label="Signed Out">
The platform is configured to authorize any request with `User-Agent:
anonymous` in place of validating the oidc id token.
💡 Take a moment to click the Signed In tab, I don't want you to miss how
cool `$(holos token)` is.
```bash
curl -A anonymous https://httpbin.holos.localhost/dump/request
```
</TabItem>
</Tabs>
You should receive a response showing the request headers the backend received.
:::tip
Note how the platform secures both web browser and command line api access to
the backend httpbin service. httpbin itself has no authentication or
authorization functionality.
:::
## Summary
Thank you for taking the time to try out Holos. In this guide, you built the
foundation of a software development platform that:
1. Provides a unified configuration model with CUE that
- Supports unmodified Helm Charts, Kustomize Kustomizations, plain YAML.
- Provides a web form to pass top level parameters.
2. Reduces errors by eliminating the need to template unstructured text.
3. Is composable and scales down to a local machine.
4. Provides an way to safely configure broad authentication and authorization
policy.
## Next Steps
Dive deeper with the following resources that build on the foundation you have now.
1. Explore the [Rendering Process](/docs/design/rendering) in Holos.
2. Dive deeper into the [Platform Manifests](./platform-manifests) rendered in this guide.
3. Deploy [ArgoCD](/docs/guides/argocd) onto the foundation you built.
4. Deploy [Backstage](/docs/guides/backstage) as a portal to the integrated platform components.
## Clean-Up
If you'd like to clean up the resources you created in this guide, remove them
with:
```bash
k3d cluster delete workload
rm -rf holos-k3d
```

View File

@@ -0,0 +1,137 @@
# Platform Manifests
This document provides an example of how Holos uses CUE and Helm to unify and
render the platform configuration. It refers to the manifests rendered in the
[Try Holos Locally](/docs/guides/try-holos/) guide.
Take a moment to review the manifests `holos` rendered to build the platform.
### ArgoCD Application
Note the Git URL in the Platform Model is used to derive the ArgoCD
`Application` resource for all of the platform components.
```yaml
# deploy/clusters/workload/gitops/namespaces.application.gen.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: namespaces
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
project: default
source:
# highlight-next-line
path: /deploy/clusters/workload/components/namespaces
# highlight-next-line
repoURL: https://github.com/holos-run/holos-k3d.git
# highlight-next-line
targetRevision: HEAD
```
One ArgoCD `Application` resource is produced for each Holos component by
default. The CUE definition which produces the rendered output is defined in
`buildplan.cue` around line 222.
:::tip
Note how CUE does not use error-prone text templates, the language is well
specified and typed which reduces errors when unifying the configuration with
the Platform Model in the following `#Argo` definition.
:::
```cue
// buildplan.cue
// #Argo represents an argocd Application resource for each component, written
// using the #HolosComponent.deployFiles field.
#Argo: {
ComponentName: string
Application: app.#Application & {
metadata: name: ComponentName
metadata: namespace: "argocd"
spec: {
destination: server: "https://kubernetes.default.svc"
project: "default"
source: {
// highlight-next-line
path: "\(_Platform.Model.argocd.deployRoot)/deploy/clusters/\(_ClusterName)/components/\(ComponentName)"
// highlight-next-line
repoURL: _Platform.Model.argocd.repoURL
// highlight-next-line
targetRevision: _Platform.Model.argocd.targetRevision
}
}
}
// deployFiles represents the output files to write along side the component.
deployFiles: "clusters/\(_ClusterName)/gitops/\(ComponentName).application.gen.yaml": yaml.Marshal(Application)
}
```
### Helm Chart
The `cert-manger` component renders using the upstream Helm chart. The build
plan that defines the helm chart to use along with the values to provide looks
like the following.
:::tip
Holos fully supports your existing Helm charts. Consider leveraging `holos` as
an alternative to umbrella charts.
:::
```cue
// components/cert-manager/cert-manager.cue
package holos
// Produce a helm chart build plan.
(#Helm & Chart).Output
let Chart = {
Name: "cert-manager"
Version: "1.14.5"
Namespace: "cert-manager"
Repo: name: "jetstack"
Repo: url: "https://charts.jetstack.io"
// highlight-next-line
Values: {
installCRDs: true
startupapicheck: enabled: false
// Must not use kube-system on gke autopilot. GKE Warden blocks access.
// highlight-next-line
global: leaderElection: namespace: Namespace
// https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-resource-requests#min-max-requests
resources: requests: {
cpu: "250m"
memory: "512Mi"
"ephemeral-storage": "100Mi"
}
// highlight-next-line
webhook: resources: Values.resources
// highlight-next-line
cainjector: resources: Values.resources
// highlight-next-line
startupapicheck: resource: Values.resources
// https://cloud.google.com/kubernetes-engine/docs/how-to/autopilot-spot-pods
nodeSelector: {
"kubernetes.io/os": "linux"
if _ClusterName == "management" {
"cloud.google.com/gke-spot": "true"
}
}
webhook: nodeSelector: Values.nodeSelector
cainjector: nodeSelector: Values.nodeSelector
startupapicheck: nodeSelector: Values.nodeSelector
}
}
```

20
doc/md/install.md Normal file
View File

@@ -0,0 +1,20 @@
# Installation
Holos is distributed as a single file executable.
## Releases
Download `holos` from the [releases](https://github.com/holos-run/holos/releases) page and place the executable into your shell path.
## Go install
Alternatively, install directly into your go bin path using:
```shell
go install github.com/holos-run/holos/cmd/holos@latest
```
### What you'll need
- [helm](https://github.com/helm/helm/releases) to fetch and render Helm chart components.
- [kubectl](https://kubernetes.io/docs/tasks/tools/) to [kustomize](https://kustomize.io/) components.

64
doc/md/intro.md Normal file
View File

@@ -0,0 +1,64 @@
# Introduction
⚡️ Holos will help you build your **software development platform in no time.**
💸 Building a software development platform is **time consuming and expensive**. Spend more time building features for your customers and less time managing your development platform.
💥 Already have a platform? Add new features and services to your platform easily with Holos.
🧐 Holos is a platform builder. It builds a hollistic software development platform composed of best-of-breed cloud native open source projects. Holos is also a tool to make it easier to manage cloud infrastructure by providing a typed alternative to yaml templates.
## Features
Holos was built to solve two main problems:
1. Building a platform usually takes 3 engineers 6-9 months of effort. Holos provides a reference platform that enables you to deploy and customize your platform in a fraction of the time.
2. Configuration changes often cause outages. Existing tools like Helm make it difficult to understand the impact a configuration change will have. Holos provides a unique, unified configuration model powered by CUE that makes it safer and easier to roll out configuration changes.
A core principle of Holos is that organizations gain value from owning the the platform they build on. Avoid vendor lock-in, future price hikes, and expensive licensing changes by building on a solid foundation of open source, cloud native computing foundation backed projects.
The following features are built into the Holos reference platform.
:::tip
Don't see your preferred technology in the stack? Holos is designed to enable you to swap out components of the platform tech stack.
:::
- **Continuous Delivery**
- Holos builds a GitOps workflow for each application running in the platform.
- Developers push changes which are automatically deployed.
- Powered by [ArgoCD](https://argo-cd.readthedocs.io/en/stable/)
- **Identity and Access Management** (IAM)
- Holos builds a standard OIDC identity provider for you.
- Integrates with your exisitng IAM and SSO system, or works independently.
- Powerful customer identity and access management features.
- Role based access control.
- Powered by [ZITADEL](https://zitadel.com/)
- **Zero Trust**
- Authenticate and Authorize users at the platform layer instead of or in addition to the application layer.
- Integrated with observability to measure and alert about problems before customers complain.
- Powered by [Istio](https://istio.io/)
- **Observability**
- Holos collects performance and availability metrics automatically, without requiring application changes.
- Optional, deeper integration into the application layer.
- Distributed Tracing
- Logging
- Powered by Prometheus, Grafana, Loki, and OpenTelemetry.
- **Data Platform**
- Integrated management of PostgreSQL
- Automatic backups
- Automatic restore from backup
- Quickly fail over across multiple regions
- **Multi-Region**
- Holos is designed to operate in multiple regions and multiple clouds.
- Keep customer data in the region that makes the most sense for your business.
- Easily cut over from one region to another for redundancy and business continuity.
## Development Status
Holos is being actively developed by [Open Infrastructure Services](https://openinfrastructure.co). Release can be found [here](https://github.com/holos-run/holos/releases).
## Adoption
Organizations who have officially adopted Holos can be found [here](https://github.com/holos-run/holos/blob/main/ADOPTERS.md).

View File

@@ -0,0 +1,28 @@
# Local Development
This document captures notes on locally developing Holos.
Follow the steps in [Try Holos Locally](/docs/guides/try-holos), but take care
to select `Develop` tabs when creating the k3d cluster so you have a local
registry to push to.
## Apply Resources
Work will be done in the `dev-holos` namespace.
Apply the infrastructure, which should persist when tilt is started / stopped.
```bash
kubectl apply --server-side=true -f ./hack/tilt/k8s/dev-holos-infra
```
This creates the PostgresCluster, service account, etc...
## Start tilt
Tilt will build the go executable, build the container, then push it to the
local repository associated with k3d.
```bash
./hack/tilt/bin/tilt up
```

393
doc/md/quickstart/index.mdx Normal file
View File

@@ -0,0 +1,393 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Admonition from '@theme/Admonition';
# Quickstart Guide
This guide shows you the basics of how Holos. You'll deploy a Helm chart to
Kubernetes using a Component to see how Holos makes the process safer, easier,
and more consistent.
## What you'll need {#Requirements}
You'll need the following tools installed to complete this guide.
1. [holos](/docs/install) - to build the platform.
2. [helm](https://helm.sh/docs/intro/install/) - to render Holos components that
wrap upstream Helm charts.
Optionally, if you'd like to apply the rendered manifests to a real cluster
you'll need:
1. [k3d](https://k3d.io/#installation) - to provide a Kubernetes API server.
2. [OrbStack](https://docs.orbstack.dev/install) or
[Docker](https://docs.docker.com/get-docker/) - to use k3d.
3. [kubectl](https://kubernetes.io/docs/tasks/tools/) - to interact with
kubernetes.
## Install Holos
Install Holos with the following command or other methods listed on the
[Installation](/docs/install/) page.
```bash
go install github.com/holos-run/holos/cmd/holos@latest
```
## Git repository
Start by initializing an empty Git repository. Holos is designed to operate
against local files in a Git repository.
<Tabs groupId="init">
<TabItem value="command" label="Command">
```bash
mkdir holos-quickstart
cd holos-quickstart
git init
```
</TabItem>
<TabItem value="output" label="Output">
```txt
Initialized empty Git repository in /holos-quickstart/.git/
```
</TabItem>
</Tabs>
This guide assumes commands are run from the root directory of this Git
repository unless otherwise stated.
## Generate the Platform {#Generate-Platform}
Generate the Platform code in the repository root. A Platform refers to all of
the software holistically integrated to provide a software development platform
for your organization. In this guide the platform will contain a single
Component to demonstrate how the concepts fit together.
```bash
holos generate platform quickstart
```
Commit the generated platform config to the repository.
<Tabs groupId="commit-platform">
<TabItem value="command" label="Command">
```bash
git add .
git commit -m "holos generate platform quickstart - $(holos --version)"
```
</TabItem>
<TabItem value="output" label="Output">
```txt
[main (root-commit) 0b17b7f] holos generate platform quickstart
213 files changed, 72349 insertions(+)
...
```
</TabItem>
</Tabs>
## Generate a Component {#generate-component}
The platform you generated is empty. Generate the CUE code definition for a
Component that wraps the podinfo Helm chart.
<Tabs groupId="gen-podinfo">
<TabItem value="command" label="Command">
```bash
holos generate component helm podinfo
```
</TabItem>
<TabItem value="output" label="Output">
```txt
generated component
```
</TabItem>
</Tabs>
This command produces two files. A leaf `components/podinfo/podinfo.gen.cue`
file, and a root `podinfo.gen.cue` file. Holos takes advantage of the fact that
[order is irrelevant](https://cuelang.org/docs/tour/basics/order-irrelevance/)
in CUE to register the component with the Platform specification by adding a
file to the root of the Git repository in addition to defining the component
itself in the leaf component directory.
The Helm chart Component is defined in the `components/podinfo/podinfo.cue`
file, for example:
<Tabs groupId="podinfo-files">
<TabItem value="components/podinfo/podinfo.gen.cue" label="Leaf">
`components/podinfo/podinfo.gen.cue`
```cue
package holos
// Produce a helm chart build plan.
(#Helm & Chart).Output
let Chart = {
Name: "podinfo"
Version: "6.6.2"
Namespace: "default"
Repo: name: "podinfo"
Repo: url: "https://stefanprodan.github.io/podinfo"
Values: {}
}
```
</TabItem>
<TabItem value="podinfo.gen.cue" label="Root">
`podinfo.gen.cue`
```cue
package holos
// Manage podinfo on workload clusters only
for Cluster in #Fleets.workload.clusters {
#Platform: Components: "\(Cluster.name)/podinfo": {
path: "components/podinfo"
cluster: Cluster.name
}
}
```
</TabItem>
</Tabs>
In this example we're providing the minimal information about the Helm chart we
want to manage. The name, version, Kubernetes namespace to deploy into, and the
chart repository location.
This chart deploys cleanly with no values provided, but we include an empty
Values struct to illustrate how Holos improves the consistency and safety of
Helm by taking advantage the strong type checking in CUE. Shared values,
such as the organization domain name, can safely be passed to all Components
across all clusters in the Platform by defining them at the root of the
configuration.
Commit the generated component config to the repository.
<Tabs groupId="commit-component">
<TabItem value="command" label="Command">
```bash
git add .
git commit -m "holos generate component helm podinfo - $(holos --version)"
```
</TabItem>
<TabItem value="output" label="Output">
```txt
[main cc0e90c] holos generate component helm podinfo
2 files changed, 24 insertions(+)
create mode 100644 components/podinfo/podinfo.gen.cue
create mode 100644 podinfo.gen.cue
```
</TabItem>
</Tabs>
## Render the Component
Individual components can be rendered without needing to be included in a
Platform spec, useful when developing a new component.
<Tabs groupId="render-podinfo">
<TabItem value="command" label="Command">
```bash
holos render component ./components/podinfo --cluster-name=default
```
</TabItem>
<TabItem value="output" label="Output">
```txt
cached
rendered podinfo
```
</TabItem>
</Tabs>
First, the command caches the helm chart locally to speed up subsequent
renderings. Then the command executes helm to produce the output which is
written into the deploy directory.
<Tabs groupId="tree-podinfo">
<TabItem value="command" label="Command">
```bash
tree deploy
```
</TabItem>
<TabItem value="output" label="Output">
```txt
deploy
└── clusters
└── default
└── components
└── podinfo
└── podinfo.gen.yaml
5 directories, 1 file
```
</TabItem>
</Tabs>
The component is deployed to one cluster named default. The same component is
often deployed to multiple clusters, for example east and west for reliability.
:::tip
This example is equivalent to executing `helm template` on the chart and saving
the output to a file. Holos simplifies this task by making it safer and more
consistent across multiple charts.
:::
## Mix in an ArgoCD Application
So far we've seen how Holos is a convenient wrapper around Helm, but we haven't
yet seen how it makes it easier to consistently and safely manage all of the
software that goes into a platform. We'll mix in an ArgoCD
[Application][application] resource to manage the podinfo Component with GitOps.
We'll define this configuration in a way that is automatically and consistently
re-used across all Components added to the Platform in the future, including
Components which are not Helm charts.
Create a new file named `argocd.cue` in the root of your git repository with the
following contents:
<Tabs groupId="argocd-config">
<TabItem value="command" label="File: argocd.cue">
```cue
package holos
#ArgoConfig: {
Enabled: true
RepoURL: "https://example.com/holos-quickstart.git"
}
```
</TabItem>
<TabItem value="note" label="Note">
If you plan to apply the rendered output to a real cluster, change the RepoURL
to the url of the git repository you created in this guide. It is sufficient to
keep the example URL if you're getting a feel for Holos and inspecting the
rendered output without applying it to a live cluster.
</TabItem>
</Tabs>
With this file in place, render the component again.
<Tabs groupId="render-podinfo-argocd">
<TabItem value="command" label="Command">
```bash
holos render component ./components/podinfo --cluster-name=default
```
</TabItem>
<TabItem value="output" label="Output">
```txt
wrote deploy file
rendered gitops/podinfo
rendered podinfo
```
</TabItem>
</Tabs>
Holos uses the locally cached copy of the chart to render the output to improve
performance and reliability. Then, the Helm template output is rendered along
with an additional ArgoCD Application resource for GitOps in the
`podinfo.application.gen.yaml` file.
:::tip
By defining the ArgoCD configuration at the root, we again take advantage of the
fact that [order is
irrelevant](https://cuelang.org/docs/tour/basics/order-irrelevance/) in CUE.
:::
Defining the configuration at the root causes all of the leaf Components to take
on the ArgoCD configuration and render an Application resource for the
Component.
<Tabs groupId="tree-podinfo-argocd">
<TabItem value="command" label="Command">
```bash
tree deploy
```
</TabItem>
<TabItem value="output" label="Output">
```txt
deploy
└── clusters
└── default
├── components
│   └── podinfo
│   └── podinfo.gen.yaml
└── gitops
└── podinfo.application.gen.yaml
6 directories, 2 files
```
</TabItem>
</Tabs>
Note the new `podinfo.application.gen.yaml` created by enabling the ArgoCD in
the Helm component. The Application resource in the file looks like the
following.
<Tabs groupId="podinfo-application">
<TabItem value="file" label="podinfo.application.gen.yaml">
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: podinfo
namespace: argocd
spec:
destination:
server: https://kubernetes.default.svc
project: default
source:
path: ./deploy/clusters/default/components/podinfo
repoURL: https://example.com/holos-quickstart.git
targetRevision: main
```
</TabItem>
</Tabs>
:::tip
Holos will generate a similar Application resource for all additional Components
added to your Platform.
:::
Finally, add and commit the results to your platform Git repository.
<Tabs groupId="commit-argo">
<TabItem value="command" label="Command">
```bash
git add .
git commit -m "holos render component ./components/podinfo --cluster-name=default"
```
</TabItem>
<TabItem value="output" label="Output">
```txt
[main f95cef1] holos render component ./components/podinfo --cluster-name=default
3 files changed, 134 insertions(+)
create mode 100644 argocd.cue
create mode 100644 deploy/clusters/default/components/podinfo/podinfo.gen.yaml
create mode 100644 deploy/clusters/default/gitops/podinfo.application.gen.yaml
```
</TabItem>
</Tabs>
In this section we learned how Holos provides a simple way to add an ArgoCD
Application resource for the podinfo Component which wraps a Helm chart. Holos
provides consistency by managing an Application resource for every Component
added to the platform, all by defining the configuration of ArgoCD in the
`argocd.cue` file in the root of the Git repository.
## Quickstart Recap {#quickstart-recap}
In this guide we learned how to:
1. Install Holos.
2. Generate a Git repository for the Platform config.
3. Create a Component that wraps the upstream podinfo Helm Chart without modifications.
4. Render individual components.
5. Mix in an ArgoCD Application resource to every Component in the Platform.
[application]: https://argo-cd.readthedocs.io/en/stable/user-guide/application-specification/

View File

@@ -0,0 +1,3 @@
# Architecture
Coming soon.

14
doc/md/runbooks/argocd.md Normal file
View File

@@ -0,0 +1,14 @@
# ArgoCD
Create the deploy key secret in the management cluster.
```bash
tmp="$(mktemp -d)"
(cd $tmp && ssh-keygen -t ed25519 -f sshPrivateKey -m pem -C argocd -N '')
echo git@github.com:holos-run/holos-infra.git > "${tmp}/url"
holos create secret -n argocd --append-hash=false creds-holos-infra --from-file $tmp
rm -rf "$tmp"
```
When syncing the secret, the ExternalSecret needs to set the label
`argocd.argoproj.io/secret-type: repo-creds`.

31
doc/md/runbooks/deploy.md Normal file
View File

@@ -0,0 +1,31 @@
# Deployment
This document describes how deployment from `main` is configured.
1. Refer to the publish workflow.
2. Uses a SSH deploy key to:
3. Clone the holos-infra repo.
4. Write the image tag to saas/userdata/components/dev-holos-app/images.json
5. Run holos render platform ./platform
6. Commit and push the results.
7. ArgoCD takes over the rollout.
## Credentials
TODO: Lock this down more, the deploy key has too much access to the infra
repository.
```bash
mkdir -p tmp
cd tmp
ssh-keygen -t ed25519 -f holos-infra.key -m pem -C holos-run/holos -N ''
gh secret set DEPLOY_SSH_PRIVATE_KEY < holos-infra.key
gh api --method POST \
-H "Accept: application/vnd.github+json" \
/repos/holos-run/holos-infra/keys \
-f title='holos-run/holos deploy key' \
-f key="$(cat holos-infra.key.pub)" \
-F read_only=false
cd ..
rm -rf tmp
```

View File

@@ -0,0 +1,10 @@
# ZITADEL Backups
Refer to [Schedule backups](https://cloudnative-pg.io/documentation/1.23/backup/#scheduled-backups)
By default ZITADEL is backed up daily to S3. When restoring into a new cluster
of the same name, increment the revision variable to create a new blank folder
for the new cluster WAL. The cluster will not initialize unless the WAL
directory is empty.
The cluster will recovery from the previous rev.

View File

@@ -0,0 +1,3 @@
## Overview
TODO: This runbook needs to be updated to reflect the switch from PGO to CNPG.

Some files were not shown because too many files have changed in this diff Show More