Problem:
The use of google.protobuf.Any was making it awkward to work with the
data provided by the user. The structure of the form data is defined by
the platform engineer, so the intent of Any was to wrap the data in a
way we can pass over the network and persist in the database.
The escaped JSON encoding was problematic and error prone to decode on
the other end.
Solution:
Define the Platform values as a two level map with string keys, but with
protobuf message fields "sections" and "fields" respectively. Use
google.protobuf.Value from the struct package to encode the actual
value.
Result:
In TypeScript, google.protobuf.Value encodes and decodes easily to a
JSON value. On the go side, connect correctly handles the value as
well.
No more ugly error prone escaping:
```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"platform_id":"'${platformId}'"}' $host holos.v1alpha1.PlatformService.GetConfig
{
"sections": {
"org": {
"fields": {
"contactEmail": "jeff@openinfrastructure.co",
"displayName": "Open Infrastructure Services LLC",
"domain": "ois.run",
"name": "ois"
}
}
}
}
```
This return value is intended to be directly usable in the CUE code, so
we may further nest the values into a platform.spec key.
This patch changes the backend to store the platform config form
definition and the config values supplied by the form as JSON in the
database.
The gRPC API does not change with this patch, but may need to depending
on how this works and how easy it is to evolve the data model and add
features.
This patch is a work in progress wiring up the form to put the values to
the holos server using grpc.
In an effort to simplify the platform configuration, the structure is a
two level map with the top level being configuration sections and the
second level being the fields associated with the config section.
To support multiple kinds of values and field controls, the values are
serialized to JSON for rpc over the network and for storage in the
database. When they values are used, either by the UI or by the `holos
render` command, they're to be unmarshalled and in-lined into the
Platform Config data structure.
Pick back up ensuring the Platform rpc handler correctly encodes and
decodes the structure to the database.
Consider changing the config_form and config_values fields to JSON field
types in the database. It will likely make working with this a lot
easier.
With this patch we're ready to wire up the holos render command to fetch
the platform configuration and create the end to end demo.
Here's essentially what the render command will fetch and lay down as a
json file for CUE:
```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"platform_id":"018f2c4e-ecde-7bcb-8b89-27a99e6cc7a1"}' jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.GetPlatform | jq .platform.config.values
{
"sections": {
"org": {
"values": {
"contactEmail": "\"platform@openinfrastructure.co\"",
"displayName": "\"Open Infrastructure Services LLC\"",
"domain": "\"ois.run\"",
"name": "\"ois\""
}
}
}
}
```
This patch adds a /platform/:id route path to a PlatformDetail
component. The platform detail component calls the GetPlatform method
given the platform ID and renders the platform config form on the detail
tab.
The submit button is not yet wired up.
The API for adding platforms changes, allowing raw json bytes using the
RawConfig. The raw bytes are not presented on the read path though,
calling GetPlatforms provides the platform and the config form inline in
the response.
Use the `raw_config` field instead of `config` when creating the form
data.
```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" -d @ jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.AddPlatform <<EOF
{
"platform": {
"org_id": "018f27cd-e5ac-7f98-bfe1-2dbab208a48c",
"name": "bare2",
"raw_config": {
"form": "$(cue export ./forms/platform/ --out json | jq -cM | base64 -w0)"
}
}
}
EOF
```
This patch adds 4 fields to the Platform table:
1. Config Form represents the JSON FormlyFieldConfig for the UI.
2. Config CUE represents the CUE file containing a definition the
Config Values must unify with.
3. Config Definition is the CUE definition variable name used to unify
the values with the cue code. Should be #PlatformSpec in most
cases.
4. Config Values represents the JSON values provided by the UI.
The use case is the platform engineer defines the #PlatformSpec in cue,
and provides the form field config. The platform engineer then provides
1-3 above when adding or updating a Platform.
The UI then presents the form to the end user and provides values for 4
when the user submits the form.
This patch also refactors the AddPlatform method to accept a Platform
message. To do so we make the id field optional since it is server
assigned.
The patch also adds a database constraint to ensure platform names are
unique within the scope of an organization.
Results:
Note how the CUE representation of the Platform Form is exported to JSON
then converted to a base64 encoded string, which is the protobuf JSON
representation of a bytes[] value.
```
grpcurl -H "x-oidc-id-token: $(holos token)" -d @ jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.AddPlatform <<EOF
{
"platform": {
"id": "0d3dc0c0-bbc8-41f8-8c6e-75f0476509d6",
"org_id": "018f27cd-e5ac-7f98-bfe1-2dbab208a48c",
"name": "bare",
"config": {
"form": "$(cd internal/platforms/bare && cue export ./forms/platform/ --out json | jq -cM | base64 -w0)"
}
}
}
EOF
```
Note the requested platform ID is ignored.
```
{
"platforms": [
{
"id": "018f2af9-f7ba-772a-9db6-f985ece8fed1",
"timestamps": {
"createdAt": "2024-04-29T17:49:36.058379Z",
"updatedAt": "2024-04-29T17:49:36.058379Z"
},
"name": "bare",
"creator": {
"id": "018f27cd-e591-7f98-a9d2-416167282d37"
},
"config": {
"form": "eyJhcGlWZXJzaW9uIjoiZm9ybXMuaG9sb3MucnVuL3YxYWxwaGExIiwia2luZCI6IlBsYXRmb3JtRm9ybSIsIm1ldGFkYXRhIjp7Im5hbWUiOiJiYXJlIn0sInNwZWMiOnsic2VjdGlvbnMiOlt7Im5hbWUiOiJvcmciLCJkaXNwbGF5TmFtZSI6Ik9yZ2FuaXphdGlvbiIsImRlc2NyaXB0aW9uIjoiT3JnYW5pemF0aW9uIGNvbmZpZyB2YWx1ZXMgYXJlIHVzZWQgdG8gZGVyaXZlIG1vcmUgc3BlY2lmaWMgY29uZmlndXJhdGlvbiB2YWx1ZXMgdGhyb3VnaG91dCB0aGUgcGxhdGZvcm0uIiwiZmllbGRDb25maWdzIjpbeyJrZXkiOiJuYW1lIiwidHlwZSI6ImlucHV0IiwicHJvcHMiOnsibGFiZWwiOiJOYW1lIiwicGxhY2Vob2xkZXIiOiJleGFtcGxlIiwiZGVzY3JpcHRpb24iOiJETlMgbGFiZWwsIGUuZy4gJ2V4YW1wbGUnIiwicmVxdWlyZWQiOnRydWV9fSx7ImtleSI6ImRvbWFpbiIsInR5cGUiOiJpbnB1dCIsInByb3BzIjp7ImxhYmVsIjoiRG9tYWluIiwicGxhY2Vob2xkZXIiOiJleGFtcGxlLmNvbSIsImRlc2NyaXB0aW9uIjoiRE5TIGRvbWFpbiwgZS5nLiAnZXhhbXBsZS5jb20nIiwicmVxdWlyZWQiOnRydWV9fSx7ImtleSI6ImRpc3BsYXlOYW1lIiwidHlwZSI6ImlucHV0IiwicHJvcHMiOnsibGFiZWwiOiJEaXNwbGF5IE5hbWUiLCJwbGFjZWhvbGRlciI6IkV4YW1wbGUgT3JnYW5pemF0aW9uIiwiZGVzY3JpcHRpb24iOiJEaXNwbGF5IG5hbWUsIGUuZy4gJ0V4YW1wbGUgT3JnYW5pemF0aW9uJyIsInJlcXVpcmVkIjp0cnVlfX0seyJrZXkiOiJjb250YWN0RW1haWwiLCJ0eXBlIjoiaW5wdXQiLCJwcm9wcyI6eyJsYWJlbCI6IkNvbnRhY3QgRW1haWwiLCJwbGFjZWhvbGRlciI6InBsYXRmb3JtLXRlYW1AZXhhbXBsZS5jb20iLCJkZXNjcmlwdGlvbiI6IlRlY2huaWNhbCBjb250YWN0IGVtYWlsIGFkZHJlc3MiLCJyZXF1aXJlZCI6dHJ1ZX19XX1dfX0K"
}
}
]
}
```
This patch adds a basic AddPlatform method that adds a platform with a
name and a display name.
Next steps are to add fields for the Platform Config Form definition and
the Platform Config values submitted from the form.
Next step: AddPlatform
Also consider extracting the queries to get the requested org_id to a
helper function. This will likely eventually move to an interceptor
because every request is org scoped and needs authorization checks
against the org.
```
grpcurl -H "x-oidc-id-token: $(holos token)" -d '{"org_id":"018f27cd-e5ac-7f98-bfe1-2dbab208a48c"}' jeff.app.dev.k2.holos.run:443 holos.v1alpha1.PlatformService.GetPlatforms
```
This patch adds an organization "selector" that's really just a place
holder. The active organization is the last element in the list
returned by the GetCallerOrganizations method for now.
The purpose is to make sure we have the structure in place for more than
one organizations without needing to implement full support for the
feature at this early stage.
The Angular frontend is expected to call the activeOrg() method of the
OrganizationService. In the future this could store the state of which
organization the user has selected. The purpose is to return an org id
to send as a request parameter for other requests.
Note this patch also implements refresh behavior. The list of orgs is
fetched once on application load. If there is no user, or the user has
zero orgs, the user is created and an organization is added with them as
an owner. This is accompished using observable pipes.
The pipe is tied to a refresh behavior. Clicking the org button
triggers the refresh behavior, which executes the pipe again and
notifies all subscribers.
This works quite well and should be idiomatic angular / rxjs. Clicking
the button automatically updates the UI after making the necessary API
calls.
This patch simplifies the user and organization registration and query
for the UI. The pattern clients are expected to follow is to create if
the get fails. For example, the following pseudo-go-code is the
expected calling convention:
var entity *ent.User
entity, err := Get()
if err != nil {
if ent.MaskNotFound(err) == nil {
entity = Create()
} else {
return err
}
}
return entity
This patch adds the following service methods. For initial
registration, all input data comes from the id token claims of the
authenticated user.
```
❯ grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 list | xargs -n1 grpcurl -H "x-oidc-id-token: $(holos token)" jeff.app.dev.k2.holos.run:443 list
holos.v1alpha1.OrganizationService.CreateCallerOrganization
holos.v1alpha1.OrganizationService.GetCallerOrganizations
holos.v1alpha1.UserService.CreateCallerUser
holos.v1alpha1.UserService.GetCallerClaims
holos.v1alpha1.UserService.GetCallerUser
```
This copies the login command from the previous holos cli. Wire
dependency injection and all the rest of the unnecessary stuff from
kubelogin are removed, streamlined down into a single function that
takes a few oidc related parameters.
This will need to be extracted out into an infrastructure service so
multiple other command line tools can easily re-use it and get the ID
token into the x-oidc-id-token header.
Add Tilt back from holos server
Note with this patch the ec-creds.yaml file needs to be applied to the
provisioner and an external secret used to sync the image pull creds.
With this patch the dev instance is accessible behind the auth proxy.
pgAdmin also works from the Tilt UI.
https://jeff.holos.dev.k2.ois.run/app/start
Remove the server.Config struct, not needed. Remove the app struct and
move the configuration to the main holos.Config.ServerConfig.
Add flags specific to server configuration.
With this patch logging is simplified. Subcommands have a handle on the
top level holos.Config and can get a fully configured logger from
cfg.Logger() after flag parsing happens.
Disambiguate the term `core` which should mean the core domain. The app
is a supporting domain concerned with logging and configuration
initialization early in the life cycle.
