Compare commits

..

1 Commits

Author SHA1 Message Date
Jeff McCune
18233f3b33 compile: add basic structure of holos compile command
This command reads Component objects from a reader, exports a BuildPlan
from CUE, then marshals the result to the writer.

The purpose is to run concurrent instances of CUE to speed up build plan
generation prior to task execution.
2025-04-08 17:00:26 -07:00
11 changed files with 539 additions and 195 deletions

3
.gitignore vendored
View File

@@ -16,5 +16,4 @@ node_modules/
# nix
/.direnv/
result
.aider*
result

View File

@@ -1,110 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Holos is a configuration management tool for Kubernetes that implements the rendered manifests pattern using CUE. It unifies Helm charts, Kustomize bases, and raw Kubernetes manifests into a single, declarative pipeline.
### Core Flow
```
Platform → Components → BuildPlan → Generators → Transformers → Validators → Manifests
```
## Key Commands
```bash
# Development
make build # Build the binary
make test # Run all tests
make fmt # Format Go code
make lint # Run linters
make coverage # Generate coverage report
# Documentation
make update-docs # Update generated docs
make website # Build the documentation website
# Usage
holos render platform # Render entire platform
holos render component # Render single component
holos show buildplans # Show build plans
holos init platform # Initialize new platform
```
## Architecture
### Directory Structure
- `/api/` - API definitions (v1alpha5 stable, v1alpha6 in development)
- `/cmd/` - CLI entry point
- `/internal/cli/` - Command implementations
- `/internal/component/` - Component handling logic
- `/internal/platform/` - Platform handling logic
- `/internal/generate/` - Code generation
### Key Files
- `/internal/cli/render/render.go` - Core render logic
- `/internal/component/component.go` - Component processing
- `/api/core/v1alpha*/types.go` - API type definitions
### Component Types
1. **Helm** - Wraps Helm charts
2. **Kustomize** - Wraps Kustomize bases
3. **Kubernetes** - Raw Kubernetes manifests
## CUE Patterns
Components are defined in CUE:
```cue
package holos
holos: Component.BuildPlan
Component: #Helm & {
Name: "example"
Chart: {
version: "1.0.0"
repository: {
name: "example"
url: "https://charts.example.com"
}
}
}
```
## Testing
- Unit tests: `*_test.go` files colocated with source
- Integration tests: `/cmd/holos/tests/`
- Example platforms: `/internal/testutil/fixtures/`
- Run single test: `go test -run TestName ./path/to/package`
## Development Patterns
1. Error handling: Use `internal/errors/` types, wrap with context
2. Logging: Use structured `slog`, get logger with `logger.FromContext(ctx)`
3. CLI commands: Follow Cobra patterns in `/internal/cli/`
4. CUE formatting: Always run `cue fmt` on CUE files
5. Develop against v1alpha6 packages.
6. Commits: Use the package name as the first word in the commit, lower case. Commit without asking permission.
## Version Management
- Version files: `/version/embedded/{major,minor,patch}`
- Bump version: `make bump`
- API versions: v1alpha5 (stable), v1alpha6 (development)
## Key Concepts
- **Platform**: Top-level configuration containing all components
- **Component**: Unit of configuration (Helm/Kustomize/Kubernetes)
- **BuildPlan**: Instructions for building a component
- **Generator**: Creates manifests (Helm, Kustomize, etc.)
- **Transformer**: Modifies generated manifests
- **Validator**: Validates final manifests
## Resources
- Tutorials: `/doc/md/tutorial/`
- Platform templates: `/internal/generate/platforms/`
- Test fixtures: `/internal/testutil/fixtures/`

View File

@@ -1,52 +0,0 @@
# Compare Buildplans
Use the `holos compare buildplans <f1> <f2>` command to compare two BuildPlan
Files. Useful to ensure different configuration versions produce the same
results.
The `holos show buildplans` command writes a BuildPlan File to standard output.
A BuildPlan File is a yaml encoded stream of BuildPlan objects.
## User Requirements
1. `holos compare buildplans before.yaml after.yaml` must return exit code 1 when after.yaml contains fields (recursively) not present in before.yaml
2. `holos compare buildplans before.yaml after.yaml --backwards-compatible` must return exit code 0 when after.yaml contains fields (recursively) not present in before.yaml
## Behavior Specification
BuildPlan File f1 is equivalent to f2 when:
1. f1 and f2 have an equal number of BuildPlan objects.
2. each object in f1 is equivalent to exactly one unique object in f2.
Two BuildPlans, bp1 and bp2, are equivalent when:
1. All field values in bp1 are equivalent to the same field in bp2.
2. Both 1 and 2 apply to nested objects, recursively.
3. Field f is equivalent when bp1.f exactly equals bp2.f, except for:
3.1. Objects in the spec.artifacts list may appear in any arbitrary order.
3.2. The ordering of keys does not matter.
4. Backwards compatibility behavior (controlled by isBackwardsCompatible):
- When false: bp2 and bp1 must have exactly the same fields
- When true: bp2 may have additional fields that don't exist in bp1
(e.g., new features added in a newer version)
Example:
bp1 has {name: "x", version: "1.0"}
bp2 has {name: "x", version: "1.0", newFeature: "enabled"}
This comparison passes when isBackwardsCompatible=true
5. Fields in bp1 must always be present in bp2 (regardless of backwards
compatibility mode).
6. List type fields with a null value are equivalent to:
6.1. null values
6.2. empty values ([])
6.2. a missing field
A BuildPlan File is valid when:
1. Two or more identical objects exist in the same file. They must be
treated as unique objects when comparing BuildPlan Files
2. Two objects may have the same value for the metadata.name field.
3. The kind field of all objects in the file stream is "BuildPlan"
## Implementation Guidance
1. Implement a stub Comparer struct in the internal/compare package.
2. Implement a stub Comparer.BuildPlans() method.
3. Write test cases for each item in the Behavior Specification section. Use a table test approach that loads each test case from a subdirectory and reads the test case data from a `testcase.json` file. The json file should have an exitCode, name, msg, file1 and file2 fields. file1 is "before.yaml" and file2 is "after.yaml".
4. Modify the Comparer.BuildPlans() method to satisfy each test case.
5. Using the existing commands as an example, wire up the command line to the compare package.

16
go.mod
View File

@@ -13,7 +13,7 @@ require (
github.com/rogpeppe/go-internal v1.13.2-0.20241226121412-a5dc8ff20d0a
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.10.0
github.com/stretchr/testify v1.9.0
golang.org/x/sync v0.10.0
golang.org/x/tools v0.29.0
gopkg.in/yaml.v3 v3.0.1
@@ -33,7 +33,7 @@ require (
github.com/Masterminds/sprig/v3 v3.3.0 // indirect
github.com/Masterminds/squirrel v1.5.4 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/ProtonMail/go-crypto v1.1.3 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c // indirect
github.com/VividCortex/ewma v1.2.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
@@ -52,7 +52,7 @@ require (
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/cli v26.1.4+incompatible // indirect
github.com/docker/distribution v2.8.3+incompatible // indirect
github.com/docker/docker v27.1.1+incompatible // indirect
github.com/docker/docker v27.0.0+incompatible // indirect
github.com/docker/docker-credential-helpers v0.8.2 // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/go-metrics v0.0.1 // indirect
@@ -67,8 +67,8 @@ require (
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
github.com/go-errors/errors v1.4.2 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.6.0 // indirect
github.com/go-git/go-git/v5 v5.13.0 // indirect
github.com/go-git/go-billy/v5 v5.5.0 // indirect
github.com/go-git/go-git/v5 v5.11.0 // indirect
github.com/go-gorp/gorp/v3 v3.1.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
@@ -146,10 +146,10 @@ require (
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sagikazarmark/locafero v0.4.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/sergi/go-diff v1.3.1 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/skeema/knownhosts v1.3.0 // indirect
github.com/skeema/knownhosts v1.2.1 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.7.0 // indirect
@@ -169,7 +169,7 @@ require (
go.opentelemetry.io/otel/trace v1.28.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.32.0 // indirect
golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8 // indirect
golang.org/x/mod v0.22.0 // indirect
golang.org/x/net v0.34.0 // indirect
golang.org/x/oauth2 v0.25.0 // indirect

73
go.sum
View File

@@ -29,8 +29,8 @@ github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERo
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/Microsoft/hcsshim v0.11.7 h1:vl/nj3Bar/CvJSYo7gIQPyRWc9f3c6IeSNavBTSZNZQ=
github.com/Microsoft/hcsshim v0.11.7/go.mod h1:MV8xMfmECjl5HdO7U/3/hFVnkmSBjAjmA09d4bExKcU=
github.com/ProtonMail/go-crypto v1.1.3 h1:nRBOetoydLeUb4nHajyO2bKqMLfWQ/ZPwkXqXxPxCFk=
github.com/ProtonMail/go-crypto v1.1.3/go.mod h1:rA3QumHc/FZ8pAHreoekgiAbzpNsfQAosU5td4SnOrE=
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c h1:kMFnB0vCcX7IL/m9Y5LO+KQYv+t1CQOiFe6+SV2J7bE=
github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c/go.mod h1:EjAoLdwvbIOoOQr3ihjnSoLZRtE8azugULFRteWMNc0=
github.com/Shopify/logrus-bugsnag v0.0.0-20171204204709-577dee27f20d h1:UrqY+r/OJnIp5u0s1SbQ8dVfLCZJsnvazdBP5hS4iRs=
github.com/Shopify/logrus-bugsnag v0.0.0-20171204204709-577dee27f20d/go.mod h1:HI8ITrYtUY+O+ZhtlqUnD8+KwNPOyugEhfP9fdUIaEQ=
github.com/VividCortex/ewma v1.1.1/go.mod h1:2Tkkvm3sRDVXaiyucHiACn4cqf7DpdyLvmxzcbUokwA=
@@ -58,6 +58,7 @@ github.com/bugsnag/osext v0.0.0-20130617224835-0dd3f918b21b h1:otBG+dV+YK+Soembj
github.com/bugsnag/osext v0.0.0-20130617224835-0dd3f918b21b/go.mod h1:obH5gd0BsqsP2LwDJ9aOkm/6J86V6lyAXCoQWGw3K50=
github.com/bugsnag/panicwrap v0.0.0-20151223152923-e2c28503fcd0 h1:nvj0OLI3YqYXer/kZD8Ri1aaunCxIEsOst1BVJswV0o=
github.com/bugsnag/panicwrap v0.0.0-20151223152923-e2c28503fcd0/go.mod h1:D/8v3kj0zr8ZAKg1AQ6crr+5VwKN5eIywRkfhyM/+dE=
github.com/bwesterb/go-ristretto v1.2.3/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chai2010/gettext-go v1.0.2 h1:1Lwwip6Q2QGsAdl/ZKPCwTe9fe0CjlUbqj5bFNSjIRk=
@@ -66,6 +67,7 @@ github.com/cheggaaa/pb v2.0.7+incompatible/go.mod h1:pQciLPpbU0oxA0h+VJYYLxO+XeD
github.com/cheggaaa/pb/v3 v3.0.4/go.mod h1:7rgWxLrAUcFMkvJuv09+DYi7mMUYi8nO9iOWcvGJPfw=
github.com/cheggaaa/pb/v3 v3.0.8 h1:bC8oemdChbke2FHIIGy9mn4DPJ2caZYQnfbRqwmdCoA=
github.com/cheggaaa/pb/v3 v3.0.8/go.mod h1:UICbiLec/XO6Hw6k+BHEtHeQFzzBH4i2/qk/ow1EJTA=
github.com/cloudflare/circl v1.3.3/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
github.com/cloudflare/circl v1.3.7 h1:qlCDlTPz2n9fu58M0Nh1J/JzcFpfgkFHHX3O35r5vcU=
github.com/cloudflare/circl v1.3.7/go.mod h1:sRTcRWXGLrKw6yIGJ+l7amYJFfAXbZG0kBSc8r4zxgA=
github.com/cockroachdb/apd/v3 v3.2.1 h1:U+8j7t0axsIgvQUqthuNm82HIrYXodOV2iWLWtEaIwg=
@@ -99,8 +101,8 @@ github.com/docker/cli v26.1.4+incompatible h1:I8PHdc0MtxEADqYJZvhBrW9bo8gawKwwen
github.com/docker/cli v26.1.4+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/distribution v2.8.3+incompatible h1:AtKxIZ36LoNK51+Z6RpzLpddBirtxJnzDrHLEKxTAYk=
github.com/docker/distribution v2.8.3+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
github.com/docker/docker v27.1.1+incompatible h1:hO/M4MtV36kzKldqnA37IWhebRA+LnqqcqDja6kVaKY=
github.com/docker/docker v27.1.1+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker v27.0.0+incompatible h1:JRugTYuelmWlW0M3jakcIadDx2HUoUO6+Tf2C5jVfwA=
github.com/docker/docker v27.0.0+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker-credential-helpers v0.8.2 h1:bX3YxiGzFP5sOXWc3bTPEXdEaZSeVMrFgOr3T+zrFAo=
github.com/docker/docker-credential-helpers v0.8.2/go.mod h1:P3ci7E3lwkZg6XiHdRKft1KckHiO9a2rNtyFbZ/ry9M=
github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c=
@@ -111,8 +113,8 @@ github.com/docker/go-metrics v0.0.1 h1:AgB/0SvBxihN0X8OR4SjsblXkbMvalQ8cjmtKQ2rQ
github.com/docker/go-metrics v0.0.1/go.mod h1:cG1hvH2utMXtqgqqYE9plW6lDxS3/5ayHzueweSI3Vw=
github.com/docker/libtrust v0.0.0-20150114040149-fa567046d9b1 h1:ZClxb8laGDf5arXfYcAtECDFgAgHklGI8CxgjHnXKJ4=
github.com/docker/libtrust v0.0.0-20150114040149-fa567046d9b1/go.mod h1:cyGadeNEkKy96OOhEzfZl+yxihPEzKnqJwvfuSUqbZE=
github.com/elazarl/goproxy v1.2.1 h1:njjgvO6cRG9rIqN2ebkqy6cQz2Njkx7Fsfv/zIZqgug=
github.com/elazarl/goproxy v1.2.1/go.mod h1:YfEbZtqP4AetfO6d40vWchF3znWX7C7Vd6ZMfdL8z64=
github.com/elazarl/goproxy v0.0.0-20230808193330-2592e75ae04a h1:mATvB/9r/3gvcejNsXKSkQ6lcIaNec2nyfOdlTBR2lU=
github.com/elazarl/goproxy v0.0.0-20230808193330-2592e75ae04a/go.mod h1:Ro8st/ElPeALwNFlcTpWmkr6IoMFfkjXAvTHpevnDsM=
github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/emicklei/proto v1.13.4 h1:myn1fyf8t7tAqIzV91Tj9qXpvyXXGXk8OS2H6IBSc9g=
@@ -141,18 +143,18 @@ github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nos
github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E=
github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ=
github.com/gliderlabs/ssh v0.3.8 h1:a4YXD1V7xMF9g5nTkdfnja3Sxy1PVDCj1Zg4Wb8vY6c=
github.com/gliderlabs/ssh v0.3.8/go.mod h1:xYoytBv1sV0aL3CavoDuJIQNURXkkfPA/wxQ1pL1fAU=
github.com/gliderlabs/ssh v0.3.5 h1:OcaySEmAQJgyYcArR+gGGTHCyE7nvhEMTlYY+Dp8CpY=
github.com/gliderlabs/ssh v0.3.5/go.mod h1:8XB4KraRrX39qHhT6yxPsHedjA08I/uBVwj4xC+/+z4=
github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI=
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376/go.mod h1:an3vInlBmSxCcxctByoQdvwPiA7DTK7jaaFDBTtu0ic=
github.com/go-git/go-billy/v5 v5.6.0 h1:w2hPNtoehvJIxR00Vb4xX94qHQi/ApZfX+nBE2Cjio8=
github.com/go-git/go-billy/v5 v5.6.0/go.mod h1:sFDq7xD3fn3E0GOwUSZqHo9lrkmx8xJhA0ZrfvjBRGM=
github.com/go-git/go-billy/v5 v5.5.0 h1:yEY4yhzCDuMGSv83oGxiBotRzhwhNr8VZyphhiu+mTU=
github.com/go-git/go-billy/v5 v5.5.0/go.mod h1:hmexnoNsr2SJU1Ju67OaNz5ASJY3+sHgFRpCtpDCKow=
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399 h1:eMje31YglSBqCdIqdhKBW8lokaMrL3uTkpGYlE2OOT4=
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399/go.mod h1:1OCfN199q1Jm3HZlxleg+Dw/mwps2Wbk9frAWm+4FII=
github.com/go-git/go-git/v5 v5.13.0 h1:vLn5wlGIh/X78El6r3Jr+30W16Blk0CTcxTYcYPWi5E=
github.com/go-git/go-git/v5 v5.13.0/go.mod h1:Wjo7/JyVKtQgUNdXYXIepzWfJQkUEIGvkvVkiXRR/zw=
github.com/go-git/go-git/v5 v5.11.0 h1:XIZc1p+8YzypNr34itUfSvYJcv+eYdTnTvOZ2vD3cA4=
github.com/go-git/go-git/v5 v5.11.0/go.mod h1:6GFcX2P3NM7FPBfpePbpLd21XxsgdAt+lKqXmCUiUCY=
github.com/go-gorp/gorp/v3 v3.1.0 h1:ItKF/Vbuj31dmV4jxA1qblpSwkl9g1typ24xoe70IGs=
github.com/go-gorp/gorp/v3 v3.1.0/go.mod h1:dLEjIyyRNiXvNZ8PSmzpt1GsWAUK8kjVhEpjH8TixEw=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
@@ -358,8 +360,8 @@ github.com/onsi/ginkgo/v2 v2.19.0 h1:9Cnnf7UHo57Hy3k6/m5k3dRfGTMXGvxhHFvkDTCTpvA
github.com/onsi/ginkgo/v2 v2.19.0/go.mod h1:rlwLi9PilAFJ8jCg9UE1QP6VBpd6/xj3SRC0d6TU0To=
github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/onsi/gomega v1.34.1 h1:EUMJIKUjM8sKjYbtxQI9A4z2o+rruxnzNvpknOXie6k=
github.com/onsi/gomega v1.34.1/go.mod h1:kU1QgUvBDLXBJq618Xvm2LUX6rSAfRaFRTcdOeDLwwY=
github.com/onsi/gomega v1.33.1 h1:dsYjIxxSR755MDmKVsaFQTE22ChNBcuuTWgkUDSubOk=
github.com/onsi/gomega v1.33.1/go.mod h1:U4R44UsT+9eLIaYRB2a5qajjtQYn0hauxvRm16AVYg0=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug=
@@ -421,16 +423,16 @@ github.com/sagikazarmark/locafero v0.4.0 h1:HApY1R9zGo4DBgr7dqsTH/JJxLTTsOt7u6ke
github.com/sagikazarmark/locafero v0.4.0/go.mod h1:Pe1W6UlPYUk/+wc/6KFhbORCfqzgYEpgQ3O5fPuL3H4=
github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE=
github.com/sagikazarmark/slog-shim v0.1.0/go.mod h1:SrcSrq8aKtyuqEI1uvTDTK1arOWRIczQRv+GVI1AkeQ=
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 h1:n661drycOFuPLCN3Uc8sB6B/s6Z4t2xvBgU1htSHuq8=
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
github.com/sergi/go-diff v1.3.1 h1:xkr+Oxo4BOQKmkn/B9eMK0g5Kg/983T9DqqPHwYqD+8=
github.com/sergi/go-diff v1.3.1/go.mod h1:aMJSSKb2lpPvRNec0+w3fl7LP9IOFzdc9Pa4NFbPK1I=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/skeema/knownhosts v1.3.0 h1:AM+y0rI04VksttfwjkSTNQorvGqmwATnvnAHpSgc0LY=
github.com/skeema/knownhosts v1.3.0/go.mod h1:sPINvnADmT/qYH1kfv+ePMmOBTH6Tbl7b5LvTDjFK7M=
github.com/skeema/knownhosts v1.2.1 h1:SHWdIUa82uGZz+F+47k8SY4QhhI291cXCpopT1lK2AQ=
github.com/skeema/knownhosts v1.2.1/go.mod h1:xYbVRSPxqBZFrdmDyMmsOs+uX1UZC3nTN3ThzgDxUwo=
github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0=
github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8=
@@ -452,8 +454,8 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/tetratelabs/wazero v1.6.0 h1:z0H1iikCdP8t+q341xqepY4EWvHEw8Es7tlqiVzlP3g=
@@ -475,6 +477,7 @@ github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ=
github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yvasiyarov/go-metrics v0.0.0-20140926110328-57bccd1ccd43 h1:+lm10QQTNSBd8DVTNGHx7o/IKu9HYDvLMffDhbyLccI=
github.com/yvasiyarov/go-metrics v0.0.0-20140926110328-57bccd1ccd43/go.mod h1:aX5oPXxHm3bOH+xeAttToC8pqch2ScQN/JoXYupl6xs=
github.com/yvasiyarov/gorelic v0.0.0-20141212073537-a9bba5b9ab50 h1:hlE8//ciYMztlGpl/VA+Zm1AcTPHYkHJPbHqE6WJUXE=
@@ -499,13 +502,18 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.3.1-0.20221117191849-2c476679df9a/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0LeHDbnYEryqj5Q1ug8=
golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY=
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8 h1:yixxcjnhBmY0nkL253HFVIm0JsFHwrHdT3Yh6szTnfY=
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8/go.mod h1:jj3sYF3dwk5D+ghuXyeI3r5MFf+NT2An6/9dOA95KSI=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.22.0 h1:D4nJWe9zXqHOmWqj4VMOJhvzj7bEZg4wEYa759z1pH4=
golang.org/x/mod v0.22.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -516,7 +524,12 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
golang.org/x/net v0.34.0 h1:Mb7Mrk043xzHgnRM88suvJFwzVrRfHEHJEl5/71CKw0=
golang.org/x/net v0.34.0/go.mod h1:di0qlW3YNM5oh6GqDGQr92MyTozJPmybPK4Ev/Gm31k=
golang.org/x/oauth2 v0.25.0 h1:CY4y7XT9v0cRI9oupztF8AgiIu99L/ksR/Xp/6jrZ70=
@@ -527,6 +540,8 @@ golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -555,17 +570,29 @@ golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.29.0 h1:TPYlXGxvx1MGTn2GiZDhnjPA9wZzZeGKHHmKhHYvgaU=
golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg=
golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk=
@@ -575,6 +602,8 @@ golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtn
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.29.0 h1:Xx0h3TtM9rzQpQuR4dKLrdglAmCEN5Oi+P74JdhdzXE=
golang.org/x/tools v0.29.0/go.mod h1:KMQVMRsVxU6nHCFXrBPhDB8XncLNLM0lIy/F14RP588=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

View File

@@ -13,7 +13,7 @@ func New(name string) *cobra.Command {
cmd := &cobra.Command{
Use: name,
Short: name,
Version: version.Version,
Version: version.GetVersion(),
Args: cobra.NoArgs,
CompletionOptions: cobra.CompletionOptions{
HiddenDefaultCmd: true,

27
internal/cli/compile.go Normal file
View File

@@ -0,0 +1,27 @@
package cli
import (
_ "embed"
"github.com/holos-run/holos/internal/cli/command"
"github.com/holos-run/holos/internal/compile"
"github.com/holos-run/holos/internal/errors"
"github.com/spf13/cobra"
)
//go:embed compile.txt
var compileLong string
// NewCompileCmd returns a new compile command.
func NewCompileCmd() *cobra.Command {
cmd := command.New("compile")
cmd.Short = "Compile Components (stdin) to BuildPlans (stdout) using CUE"
cmd.Long = compileLong
cmd.Args = cobra.NoArgs
cmd.RunE = func(cmd *cobra.Command, args []string) error {
c := compile.New()
ctx := cmd.Root().Context()
return errors.Wrap(c.Run(ctx))
}
return cmd
}

12
internal/cli/compile.txt Normal file
View File

@@ -0,0 +1,12 @@
Reads a stream of JSON-encoded Component objects from standard input. For each
Component, builds a CUE instance then exports a BuildPlan. Each resulting
BuildPlan is written as a JSON-encoded object to standard output.
This command encapsulates cue export for concurrent use. It may be used directly
by piping the output of holos show platform through jq to select the components
list, then to stdin of this command. This command is most often invoked 'holos
render platform' to run concurrent cue exports safely.
For example:
holos show platform --format=json | jq '.spec.components[]' | holos compile

View File

@@ -76,6 +76,9 @@ func New(cfg *holos.Config) *cobra.Command {
// Show
rootCmd.AddCommand(NewShowCmd(platform.NewConfig()))
// Compile
rootCmd.AddCommand(NewCompileCmd())
return rootCmd
}

View File

@@ -0,0 +1,78 @@
// Package compile compiles BuildPlan resources by reading json encoded data
// from a reader, unmarshaling the data into a Component, building a CUE
// instance injecting the Component as a tag, then exporting a BuildPlan and
// marshalling the result to a writer represented as a stream of json objects.
// Each input component maps to one output json object in the stream.
package compile
import (
"context"
"encoding/json"
"fmt"
"io"
"log/slog"
"os"
"github.com/holos-run/holos/internal/errors"
"github.com/holos-run/holos/internal/holos"
)
// New returns a new BuildPlan Compiler.
func New(opts ...Option) *Compiler {
c := &Compiler{
r: os.Stdin,
w: os.Stdout,
}
for _, o := range opts {
o(c)
}
return c
}
type Option func(c *Compiler)
func WithReader(r io.Reader) Option {
return func(c *Compiler) { c.r = r }
}
func WithWriter(w io.Writer) Option {
return func(c *Compiler) { c.w = w }
}
type Compiler struct {
r io.Reader
w io.Writer
}
func (c *Compiler) Run(ctx context.Context) error {
dec := json.NewDecoder(c.r)
enc := json.NewEncoder(c.w)
slog.DebugContext(ctx, "entering read loop")
for {
select {
case <-ctx.Done():
return errors.Wrap(ctx.Err())
default:
}
var raw json.RawMessage
err := dec.Decode(&raw)
if err == io.EOF {
slog.DebugContext(ctx, "received: eof", "eof", true)
return nil
}
var meta holos.TypeMeta
err = json.Unmarshal(raw, &meta)
if err != nil {
return errors.Format("could not unmarshal input: %w", err)
}
slog.DebugContext(ctx, fmt.Sprintf("received: %+v", meta), "meta", meta)
err = enc.Encode(raw)
if err != nil {
return errors.Format("could not marshal output: %w", err)
}
}
}

View File

@@ -0,0 +1,358 @@
//go:build gemini
package compile
import (
"bufio"
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"runtime" // To get number of CPUs
"golang.org/x/sync/errgroup"
)
// Placeholder types - replace with your actual definitions
type Component struct {
ID int `json:"id"` // Add an ID to correlate requests/responses if needed later
Name string `json:"name"`
Data string `json:"data"`
// Add other fields as needed
}
type BuildPlan struct {
ComponentID int `json:"componentId"` // Correlates back to Component.ID
ComponentUsed string `json:"componentUsed"`
Result map[string]interface{} `json:"result"`
Success bool `json:"success"`
WorkerPID int `json:"workerPid"` // Optional: useful for debugging
// Add other fields as needed
}
// executeReusableWorkers starts a pool of worker processes, distributes components
// to them, and collects build plans. Uses line-delimited JSON over stdio.
func executeReusableWorkers(ctx context.Context, numWorkers int, components []Component) ([]BuildPlan, error) {
if len(components) == 0 {
return nil, nil
}
if numWorkers <= 0 {
numWorkers = runtime.NumCPU() // Default to number of CPUs
}
if numWorkers > len(components) {
numWorkers = len(components) // No need for more workers than tasks
}
exePath, err := os.Executable()
if err != nil {
return nil, fmt.Errorf("failed to get executable path: %w", err)
}
// Channel for distributing components to worker goroutines
// Buffer size == len(components) to avoid blocking the sender initially
tasksChan := make(chan Component, len(components))
// Channel for collecting results from worker goroutines
// Buffer size helps prevent worker goroutines blocking if main thread is slow
resultsChan := make(chan BuildPlan, len(components))
// Use an error group to manage worker goroutines and capture the first error
g, childCtx := errgroup.WithContext(ctx)
// Start worker processes and their managing goroutines
for i := 0; i < numWorkers; i++ {
workerIndex := i // Capture loop variable
// Create the command for the worker process
// Ensure the child knows it's a worker (e.g., via "--worker" flag)
cmd := exec.CommandContext(childCtx, exePath, "--worker") // Adapt flag as needed
stdinPipe, err := cmd.StdinPipe()
if err != nil {
return nil, fmt.Errorf("worker %d: failed to get stdin pipe: %w", workerIndex, err)
}
stdoutPipe, err := cmd.StdoutPipe()
if err != nil {
_ = stdinPipe.Close()
return nil, fmt.Errorf("worker %d: failed to get stdout pipe: %w", workerIndex, err)
}
var stderrBuf bytes.Buffer
cmd.Stderr = &stderrBuf
// Start the worker process
if err := cmd.Start(); err != nil {
_ = stdinPipe.Close()
// stdoutPipe closing is less critical before Wait
return nil, fmt.Errorf("worker %d: failed to start command '%s': %w", workerIndex, exePath, err)
}
pid := cmd.Process.Pid // Get PID for logging/debugging
fmt.Printf("Parent: Started worker %d (PID: %d)\n", workerIndex, pid)
// Launch a goroutine to manage this specific worker
g.Go(func() error {
// Ensure resources are cleaned up for this worker's goroutine
defer func() {
fmt.Printf("Parent: Goroutine for worker %d (PID: %d) shutting down stdin.\n", workerIndex, pid)
// Closing stdin signals the worker to terminate its loop
stdinPipe.Close()
// Wait for the process to fully exit after stdin is closed
waitErr := cmd.Wait()
stderrContent := stderrBuf.String()
if waitErr != nil {
fmt.Fprintf(os.Stderr, "Parent: Worker %d (PID: %d) exited with error (stderr: %q): %v\n", workerIndex, pid, stderrContent, waitErr)
// Note: Returning an error here might race with errors from I/O below, errgroup handles this.
} else {
fmt.Printf("Parent: Worker %d (PID: %d) exited cleanly.\n", workerIndex, pid)
}
}()
// Use buffered I/O for efficiency with line-based protocol
writer := bufio.NewWriter(stdinPipe)
// Use bufio.Scanner for reading line-delimited output
scanner := bufio.NewScanner(stdoutPipe)
// Process tasks from the channel until it's closed and empty
for task := range tasksChan {
// Marshal component to JSON
jsonData, err := json.Marshal(task)
if err != nil {
return fmt.Errorf("worker %d (PID: %d): failed to marshal component %d: %w", workerIndex, pid, task.ID, err)
}
// Write JSON line to worker's stdin
// fmt.Printf("Parent: Sending task %d to worker %d (PID: %d)\n", task.ID, workerIndex, pid) // Debug logging
if _, err := writer.Write(jsonData); err != nil {
return fmt.Errorf("worker %d (PID: %d): failed to write component %d to stdin: %w", workerIndex, pid, task.ID, err)
}
if err := writer.WriteByte('\n'); err != nil { // Write newline delimiter
return fmt.Errorf("worker %d (PID: %d): failed to write newline to stdin for component %d: %w", workerIndex, pid, task.ID, err)
}
if err := writer.Flush(); err != nil { // Ensure data is sent
return fmt.Errorf("worker %d (PID: %d): failed to flush stdin for component %d: %w", workerIndex, pid, task.ID, err)
}
// Read line (JSON BuildPlan) from worker's stdout
if !scanner.Scan() {
// Scanner failed, check for errors or premature EOF
if err := scanner.Err(); err != nil {
return fmt.Errorf("worker %d (PID: %d): error scanning stdout after sending component %d: %w", workerIndex, pid, task.ID, err)
}
// If no scanner error, it means EOF was reached unexpectedly
return fmt.Errorf("worker %d (PID: %d): unexpected EOF reading stdout after sending component %d", workerIndex, pid, task.ID)
}
line := scanner.Bytes() // Get the line bytes
// Unmarshal the BuildPlan
var plan BuildPlan
if err := json.Unmarshal(line, &plan); err != nil {
return fmt.Errorf("worker %d (PID: %d): failed to unmarshal build plan (line: %q): %w", workerIndex, pid, string(line), err)
}
plan.WorkerPID = pid // Add worker PID for tracking
// Send the result back to the main goroutine
select {
case resultsChan <- plan:
// fmt.Printf("Parent: Received result for task %d from worker %d (PID: %d)\n", plan.ComponentID, workerIndex, pid) // Debug logging
case <-childCtx.Done():
return fmt.Errorf("worker %d (PID: %d): context cancelled while sending result for component %d: %w", workerIndex, pid, task.ID, childCtx.Err())
}
}
// tasksChan was closed and this worker processed all its assigned tasks
fmt.Printf("Parent: Worker %d (PID: %d) finished processing tasks.\n", workerIndex, pid)
return nil // Goroutine finished successfully
})
}
// Goroutine to distribute tasks
// This runs concurrently with the worker goroutines
go func() {
fmt.Printf("Parent: Distributing %d tasks...\n", len(components))
for i, comp := range components {
comp.ID = i // Assign a unique ID for potential correlation
select {
case tasksChan <- comp:
// Task sent
case <-childCtx.Done():
// Context cancelled before all tasks could be sent
fmt.Fprintf(os.Stderr, "Parent: Task distribution cancelled: %v\n", childCtx.Err())
// Closing tasksChan here ensures workers eventually stop asking for tasks
close(tasksChan)
return
}
}
// After sending all tasks, close the channel to signal workers
close(tasksChan)
fmt.Println("Parent: Finished distributing tasks and closed tasks channel.")
}()
// Wait for all worker goroutines to complete (or one to error out)
fmt.Println("Parent: Waiting for workers to finish...")
err = g.Wait() // Returns the first error encountered by any worker goroutine
// Close the results channel *after* all worker goroutines have finished
// This signals the final result collection step
fmt.Println("Parent: All worker goroutines finished or errored. Closing results channel.")
close(resultsChan)
// Collect all results sent by the workers
// This loop reads until resultsChan is closed
finalResults := make([]BuildPlan, 0, len(components))
for result := range resultsChan {
finalResults = append(finalResults, result)
}
fmt.Printf("Parent: Collected %d results.\n", len(finalResults))
// Return results even if there was an error, they might be partial
if err != nil {
// Log the primary error that caused the errgroup to exit
fmt.Fprintf(os.Stderr, "Parent: executeReusableWorkers returning with error: %v\n", err)
// The finalResults slice might contain results from before the error occurred
return finalResults, err
}
// Check if we got the expected number of results (only if no error occurred)
if len(finalResults) != len(components) {
return finalResults, fmt.Errorf("mismatch: expected %d results, got %d", len(components), len(finalResults))
}
return finalResults, nil // Success
}
// --- Child Worker Logic (Must be added to main.go) ---
// // Example main function incorporating the worker logic
// func main() {
// // Check if running as a worker
// if len(os.Args) > 1 && os.Args[1] == "--worker" {
// // Set GOMAXPROCS? Maybe not necessary if CUE isn't concurrent anyway.
// runWorker() // Run the dedicated worker function
// return // Important: worker exits via runWorker
// }
//
// // --- Parent Process Logic ---
// fmt.Println("Running as parent...")
// componentsToProcess := []Component{
// {Name: "CompA", Data: "data1"},
// {Name: "CompB", Data: "data2"},
// {Name: "CompC", Data: "data3"},
// {Name: "CompD", Data: "data4"},
// {Name: "CompE", Data: "data5"},
// }
//
// ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) // Example timeout
// defer cancel()
//
// // Use, for example, 2 worker processes
// buildPlans, err := executeReusableWorkers(ctx, 2, componentsToProcess)
// if err != nil {
// // Note: buildPlans might contain partial results even if err is non-nil
// fmt.Fprintf(os.Stderr, "Error executing reusable workers: %v\n", err)
// // Decide if partial results are useful or should be discarded
// // os.Exit(1) // Optionally exit
// }
//
// fmt.Printf("Parent: Received %d build plans:\n", len(buildPlans))
// // Note: Order is not guaranteed relative to input order.
// // Sort or process based on ComponentID if needed.
// sort.Slice(buildPlans, func(i, j int) bool {
// return buildPlans[i].ComponentID < buildPlans[j].ComponentID
// })
// for _, plan := range buildPlans {
// fmt.Printf(" Plan for Component %d (from worker %d): Success=%t, Result=%v\n",
// plan.ComponentID, plan.WorkerPID, plan.Success, plan.Result)
// }
// }
// // runWorker implements the logic for a child worker process.
// // It reads line-delimited JSON Components from stdin and writes
// // line-delimited JSON BuildPlans to stdout.
// func runWorker() {
// workerPID := os.Getpid()
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Starting\n", workerPID)
// // Use buffered I/O for stdin and stdout
// stdinScanner := bufio.NewScanner(os.Stdin)
// stdoutWriter := bufio.NewWriter(os.Stdout)
// defer stdoutWriter.Flush() // Ensure buffer is flushed on exit
// // Loop reading tasks line by line from stdin
// for stdinScanner.Scan() {
// line := stdinScanner.Bytes()
// if len(line) == 0 { // Skip empty lines if any occur
// continue
// }
// var comp Component
// if err := json.Unmarshal(line, &comp); err != nil {
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Failed to decode component (line: %q): %v\n", workerPID, string(line), err)
// // Decide strategy: exit? skip? For now, exit.
// os.Exit(1)
// }
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Processing component %d (%s)\n", workerPID, comp.ID, comp.Name)
// // --- Simulate CUE processing ---
// time.Sleep(time.Duration(500+rand.Intn(500)) * time.Millisecond) // Simulate variable work
// success := true
// resultData := map[string]interface{}{
// "processedData": fmt.Sprintf("Processed %s by %d", comp.Data, workerPID),
// "timestamp": time.Now().UnixNano(),
// }
// // --- End Simulation ---
// plan := BuildPlan{
// ComponentID: comp.ID, // Echo back the ID
// ComponentUsed: comp.Name,
// Result: resultData,
// Success: success,
// // WorkerPID added by parent, not needed here
// }
// // Marshal result to JSON
// planJSON, err := json.Marshal(plan)
// if err != nil {
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Failed to marshal build plan for component %d: %v\n", workerPID, comp.ID, err)
// // Decide strategy: exit? skip? For now, exit.
// os.Exit(1)
// }
// // Write JSON result line to stdout
// if _, err := stdoutWriter.Write(planJSON); err != nil {
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Failed to write build plan for component %d to stdout: %v\n", workerPID, comp.ID, err)
// os.Exit(1)
// }
// if err := stdoutWriter.WriteByte('\n'); err != nil { // Add newline delimiter
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Failed to write newline for component %d to stdout: %v\n", workerPID, comp.ID, err)
// os.Exit(1)
// }
// // Flush the buffer after each line to ensure parent receives it
// if err := stdoutWriter.Flush(); err != nil {
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Failed to flush stdout for component %d: %v\n", workerPID, comp.ID, err)
// os.Exit(1)
// }
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Finished processing component %d (%s)\n", workerPID, comp.ID, comp.Name)
// }
// // Check for scanner errors (e.g., read errors) after the loop finishes
// if err := stdinScanner.Err(); err != nil {
// // Don't report EOF as an error, it's the signal to exit cleanly
// if err != io.EOF {
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Error reading stdin: %v\n", workerPID, err)
// os.Exit(1)
// }
// }
// // EOF reached on stdin, parent closed the pipe. Exit cleanly.
// fmt.Fprintf(os.Stderr, "Worker (PID: %d): Stdin closed, exiting cleanly.\n", workerPID)
// os.Exit(0)
// }
// NOTE: Need these imports for the example usage and worker logic:
// import (
// "io"
// "math/rand"
// "sort"
// "time"
// )