mirror of
https://github.com/cozystack/cozystack.git
synced 2026-03-17 12:28:53 +00:00
Compare commits
31 Commits
feat/expos
...
tym83/secu
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ba3ecf6893 | ||
|
|
9fb9354fd2 | ||
|
|
23bc8525be | ||
|
|
3fef40e9f7 | ||
|
|
cc5ec0b7a3 | ||
|
|
38d58a77dc | ||
|
|
47dbbb9538 | ||
|
|
37050922f2 | ||
|
|
689c2a5e4a | ||
|
|
7e0a059d34 | ||
|
|
ee8533647b | ||
|
|
b3f356a5ed | ||
|
|
ffd6e628e2 | ||
|
|
22f2e4f82a | ||
|
|
39df52542b | ||
|
|
2b60c010dd | ||
|
|
f906a0d8ad | ||
|
|
ee83aaa82e | ||
|
|
f647cfd7b9 | ||
|
|
450e2e8ec4 | ||
|
|
941fb02cd1 | ||
|
|
93992ef00b | ||
|
|
f82f13bf32 | ||
|
|
f5d8c89ddf | ||
|
|
1dd27f6b23 | ||
|
|
37a6816492 | ||
|
|
1c2c9f723a | ||
|
|
d4abb86092 | ||
|
|
42778cf4b1 | ||
|
|
374e12e1c4 | ||
|
|
4dcba2fe5a |
102
SECURITY.md
Normal file
102
SECURITY.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# Security Policy
|
||||
|
||||
## Scope
|
||||
|
||||
This policy applies to the [`cozystack/cozystack`](https://github.com/cozystack/cozystack) repository and to release artifacts produced from it, including Cozystack core components, operators, packaged manifests, container images, and installation assets published by the project.
|
||||
|
||||
Cozystack integrates and ships many upstream cloud native components. If you believe a vulnerability originates in an upstream project rather than in Cozystack-specific code, packaging, defaults, or integration logic, please report it to the upstream project as well. If you are unsure, report it to Cozystack first and we will help route or coordinate the issue.
|
||||
|
||||
## Supported Versions
|
||||
|
||||
As of March 17, 2026, the Cozystack project maintains multiple release lines. Security fixes are prioritized for the latest stable release line and, when needed, backported to other supported lines.
|
||||
|
||||
| Version line | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| `v1.1.x` | Supported | Current stable release line. |
|
||||
| `v1.0.x` | Supported | Previous stable release line; receives security and important maintenance fixes. |
|
||||
| `v0.41.x` | Limited support | Legacy pre-v1 line during the v0 to v1 transition; critical security and upgrade-blocking fixes may be backported at maintainer discretion. |
|
||||
| `< v0.41` | Not supported | Please upgrade to a supported release line before requesting a security fix. |
|
||||
| `alpha`, `beta`, `rc` releases | Not supported | Pre-release builds are for testing and evaluation only. |
|
||||
|
||||
Supported versions may change over time as new release lines are cut. The authoritative source for current releases is the GitHub Releases page:
|
||||
|
||||
<https://github.com/cozystack/cozystack/releases>
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
Please do **not** report security vulnerabilities through public GitHub issues, discussions, pull requests, Telegram, Slack, or other public community channels.
|
||||
|
||||
At the moment, this repository does not publish a dedicated private security mailbox in-tree. If you need to report a vulnerability:
|
||||
|
||||
1. Contact one of the project maintainers listed in `CODEOWNERS` using an existing private channel you already have.
|
||||
2. If you do not already have a private maintainer contact, use a public community channel only to request a private contact path, without disclosing any vulnerability details.
|
||||
|
||||
Please do not include exploit details, credentials, tokens, private keys, customer data, or other sensitive material in any public message.
|
||||
|
||||
When reporting a vulnerability, please include as much of the following as possible:
|
||||
|
||||
- affected Cozystack version, tag, or commit
|
||||
- affected component or package, for example operator, API server, dashboard, installer, or a packaged system component
|
||||
- deployment environment and provider, for example bare metal, Hetzner, Oracle Cloud, or other infrastructure
|
||||
- prerequisites and exact reproduction steps
|
||||
- impact, attack scenario, and expected blast radius
|
||||
- whether authentication, tenant access, cluster-admin access, or network adjacency is required
|
||||
- known mitigations or workarounds
|
||||
- whether you believe the issue also affects an upstream dependency
|
||||
|
||||
## What to Expect
|
||||
|
||||
The maintainers will aim to:
|
||||
|
||||
- acknowledge receipt within 3 business days
|
||||
- perform an initial triage and severity assessment within 7 business days
|
||||
- keep the reporter informed as the fix and disclosure plan are developed
|
||||
|
||||
Resolution timelines depend on severity, complexity, release branch applicability, and whether coordination with upstream projects is required.
|
||||
|
||||
## Disclosure Process
|
||||
|
||||
The Cozystack project follows a coordinated disclosure model.
|
||||
|
||||
- We ask reporters to keep details private until a fix or mitigation is available and users have had a reasonable opportunity to upgrade.
|
||||
- When appropriate, maintainers may use GitHub Security Advisories or equivalent coordinated disclosure tooling to manage remediation and public disclosure.
|
||||
- If appropriate, the project may request or publish a GHSA and/or CVE as part of the disclosure process.
|
||||
- Fixes will normally be released in the supported version lines affected by the issue, subject to severity and feasibility.
|
||||
|
||||
Public disclosure will typically happen through one or more of the following:
|
||||
|
||||
- GitHub Releases and release notes
|
||||
- project changelogs and documentation updates
|
||||
- GitHub Security Advisories, when used for coordinated disclosure
|
||||
|
||||
## Project Security Practices
|
||||
|
||||
Security is part of the normal Cozystack development and release process. Current project practices include:
|
||||
|
||||
- maintainer-owned review through pull requests and `CODEOWNERS`
|
||||
- automated pull request checks, including pre-commit validation, unit tests, builds, and end-to-end testing
|
||||
- release automation with patch releases, release branches, and backport workflows
|
||||
- ongoing maintenance of packaged dependencies and platform integrations across supported release lines
|
||||
|
||||
Because Cozystack is an integration-heavy platform, some vulnerabilities may require coordination across multiple repositories or with upstream maintainers before a public fix can be released.
|
||||
|
||||
## Security Fixes and Announcements
|
||||
|
||||
Security fixes are published in normal release artifacts whenever possible. Users should monitor:
|
||||
|
||||
- GitHub Releases: <https://github.com/cozystack/cozystack/releases>
|
||||
- project changelogs in this repository
|
||||
- the Cozystack website and documentation: <https://cozystack.io>
|
||||
|
||||
## Out of Scope
|
||||
|
||||
The following are generally out of scope for private security reporting unless there is a clear Cozystack-specific impact:
|
||||
|
||||
- vulnerabilities in unsupported or end-of-life Cozystack versions
|
||||
- issues that require access already equivalent to cluster-admin, node root, or direct infrastructure administrator privileges, unless they bypass an expected Cozystack security boundary
|
||||
- vulnerabilities that exist only in an upstream dependency and are not introduced or materially worsened by Cozystack packaging, configuration, or defaults
|
||||
- requests for security best-practice advice without a concrete vulnerability
|
||||
|
||||
## Credits
|
||||
|
||||
We appreciate responsible disclosure and will credit reporters in public advisories or release notes unless anonymous disclosure is requested.
|
||||
29
docs/changelogs/v1.0.5.md
Normal file
29
docs/changelogs/v1.0.5.md
Normal file
@@ -0,0 +1,29 @@
|
||||
<!--
|
||||
https://github.com/cozystack/cozystack/releases/tag/v1.0.5
|
||||
-->
|
||||
|
||||
## Fixes
|
||||
|
||||
* **[api] Fix spurious OpenAPI post-processing errors for non-apps group versions**: The API server no longer logs false errors while generating OpenAPI specs for core and other non-`apps.cozystack.io` group versions. The post-processor now exits early when the base `Application` schemas are absent, reducing noisy startup logs without affecting application schema generation ([**@kvaps**](https://github.com/kvaps) in #2212, #2216).
|
||||
|
||||
## Documentation
|
||||
|
||||
* **[website] Add `DependenciesNotReady` troubleshooting and correct packages management build target**: Added a troubleshooting guide for packages stuck in `DependenciesNotReady`, including how to inspect operator logs and identify missing dependencies, and fixed the outdated `make image-cozystack` command to `make image-packages` in the packages management guide ([**@kvaps**](https://github.com/kvaps) in cozystack/website#450).
|
||||
|
||||
* **[website] Clarify operator-first installation order**: Reordered the platform installation guide and tutorial so users install the Cozystack operator before preparing and applying the Platform Package, matching the rest of the installation docs and reducing setup confusion during fresh installs ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#449).
|
||||
|
||||
* **[website] Add automated installation guide for Ansible**: Added end-to-end documentation for deploying Cozystack with the `cozystack.installer` Ansible collection, including inventory examples, distro-specific playbooks, configuration reference, verification steps, and explicit version pinning guidance to help operators automate installs safely ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#442).
|
||||
|
||||
* **[website] Expand CA rotation operations guide**: Completed the CA rotation documentation with separate Talos and Kubernetes certificate rotation procedures, dry-run preview steps, and post-rotation guidance for fetching updated `talosconfig` and `kubeconfig` files after certificate changes ([**@kvaps**](https://github.com/kvaps) in cozystack/website#406).
|
||||
|
||||
* **[website] Improve backup operations documentation**: Enhanced the operator backup and recovery guide with clearer Velero enablement steps, concrete provider and bucket examples, and more useful commands for inspecting backups, schedules, restores, CRD status, and logs ([**@androndo**](https://github.com/androndo) in cozystack/website#440).
|
||||
|
||||
* **[website] Add custom metrics collection guide**: Added a monitoring guide showing how tenants can expose their own Prometheus exporters through `VMServiceScrape` and `VMPodScrape`, including namespace labeling requirements, example manifests, verification steps, and troubleshooting advice ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/website#444).
|
||||
|
||||
* **[website] Document PackageSource and Package architecture**: Added a Key Concepts reference covering `PackageSource` and `Package` reconciliation flow, dependency handling, update propagation, rollback behavior, FluxPlunger recovery, and the `cozypkg` CLI for package management ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/website#445).
|
||||
|
||||
* **[website] Refresh v1 application and platform documentation**: Fixed the documentation auto-update flow and published a broad v1 documentation refresh covering newly documented applications, updated naming and navigation, virtualization and platform content updates, and reorganized versioned docs pages ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#439).
|
||||
|
||||
---
|
||||
|
||||
**Full Changelog**: https://github.com/cozystack/cozystack/compare/v1.0.4...v1.0.5
|
||||
25
docs/changelogs/v1.1.2.md
Normal file
25
docs/changelogs/v1.1.2.md
Normal file
@@ -0,0 +1,25 @@
|
||||
<!--
|
||||
https://github.com/cozystack/cozystack/releases/tag/v1.1.2
|
||||
-->
|
||||
|
||||
## Fixes
|
||||
|
||||
* **[bucket] Fix S3 Manager endpoint mismatch with COSI credentials**: The S3 Manager UI previously constructed an `s3.<tenant>.<cluster-domain>` endpoint even though COSI-issued bucket credentials point to the root-level S3 endpoint. This caused login failures with "invalid credentials" despite valid secrets. The deployment now uses the actual endpoint from `BucketInfo`, with the old namespace-based endpoint kept only as a fallback before `BucketAccess` secrets exist ([**@IvanHunters**](https://github.com/IvanHunters) in #2211, #2215).
|
||||
|
||||
* **[platform] Fix spurious OpenAPI post-processing errors on cozystack-api startup**: The OpenAPI post-processor was being invoked for non-`apps.cozystack.io` group versions where the base `Application*` schemas do not exist, producing noisy startup errors on every API server launch. It now skips those non-apps group versions gracefully instead of returning an error ([**@kvaps**](https://github.com/kvaps) in #2212, #2217).
|
||||
|
||||
## Documentation
|
||||
|
||||
* **[website] Add troubleshooting for packages stuck in `DependenciesNotReady`**: Added an operations guide that explains how to diagnose missing package dependencies in operator logs and corrected the packages management development docs to use the current `make image-packages` target ([**@kvaps**](https://github.com/kvaps) in cozystack/website#450).
|
||||
|
||||
* **[website] Reorder installation docs to install the operator before the platform package**: Updated the platform installation guide and tutorial so the setup sequence consistently installs the Cozystack operator first, then prepares and applies the Platform Package, matching the rest of the documentation set ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#449).
|
||||
|
||||
* **[website] Add automated installation guide for the Ansible collection**: Added a full guide for deploying Cozystack with the `cozystack.installer` collection, including inventory examples, distro-specific playbooks, configuration reference, and explicit version pinning guidance ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#442).
|
||||
|
||||
* **[website] Expand monitoring and platform architecture reference docs**: Added a tenant custom metrics collection guide for `VMServiceScrape` and `VMPodScrape`, and documented `PackageSource`/`Package` architecture, reconciliation flow, rollback behavior, and the `cozypkg` workflow in Key Concepts ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/website#444, cozystack/website#445).
|
||||
|
||||
* **[website] Improve operations guides for CA rotation and Velero backups**: Completed the CA rotation documentation with dry-run and post-rotation credential retrieval steps, and expanded the backup configuration guide with concrete examples, verification commands, and clearer operator procedures ([**@kvaps**](https://github.com/kvaps) in cozystack/website#406; [**@androndo**](https://github.com/androndo) in cozystack/website#440).
|
||||
|
||||
---
|
||||
|
||||
**Full Changelog**: https://github.com/cozystack/cozystack/compare/v1.1.1...v1.1.2
|
||||
@@ -1,4 +1,4 @@
|
||||
KUBERNETES_VERSION = v1.35
|
||||
KUBERNETES_VERSIONS = $(shell awk -F'"' '{print $$2}' files/versions.yaml)
|
||||
KUBERNETES_PKG_TAG = $(shell awk '$$1 == "version:" {print $$2}' Chart.yaml)
|
||||
|
||||
include ../../../hack/common-envs.mk
|
||||
@@ -15,17 +15,19 @@ update:
|
||||
image: image-ubuntu-container-disk image-kubevirt-cloud-provider image-kubevirt-csi-driver image-cluster-autoscaler
|
||||
|
||||
image-ubuntu-container-disk:
|
||||
docker buildx build images/ubuntu-container-disk \
|
||||
--build-arg KUBERNETES_VERSION=${KUBERNETES_VERSION} \
|
||||
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(KUBERNETES_VERSION)) \
|
||||
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(KUBERNETES_VERSION)-$(TAG)) \
|
||||
--cache-from type=registry,ref=$(REGISTRY)/ubuntu-container-disk:latest \
|
||||
--cache-to type=inline \
|
||||
--metadata-file images/ubuntu-container-disk.json \
|
||||
$(BUILDX_ARGS)
|
||||
echo "$(REGISTRY)/ubuntu-container-disk:$(call settag,$(KUBERNETES_VERSION))@$$(yq e '."containerimage.digest"' images/ubuntu-container-disk.json -o json -r)" \
|
||||
> images/ubuntu-container-disk.tag
|
||||
rm -f images/ubuntu-container-disk.json
|
||||
$(foreach ver,$(KUBERNETES_VERSIONS), \
|
||||
docker buildx build images/ubuntu-container-disk \
|
||||
--build-arg KUBERNETES_VERSION=$(ver) \
|
||||
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
|
||||
--tag $(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)-$(TAG)) \
|
||||
--cache-from type=registry,ref=$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver)) \
|
||||
--cache-to type=inline \
|
||||
--metadata-file images/ubuntu-container-disk-$(ver).json \
|
||||
$(BUILDX_ARGS) && \
|
||||
echo "$(REGISTRY)/ubuntu-container-disk:$(call settag,$(ver))@$$(yq e '."containerimage.digest"' images/ubuntu-container-disk-$(ver).json -o json -r)" \
|
||||
> images/ubuntu-container-disk-$(ver).tag && \
|
||||
rm -f images/ubuntu-container-disk-$(ver).json; \
|
||||
)
|
||||
|
||||
image-kubevirt-cloud-provider:
|
||||
docker buildx build images/kubevirt-cloud-provider \
|
||||
|
||||
@@ -317,11 +317,7 @@ func (w *WrappedControllerService) ControllerPublishVolume(ctx context.Context,
|
||||
"ownerReferences": []interface{}{vmiOwnerRef},
|
||||
},
|
||||
"spec": map[string]interface{}{
|
||||
"endpointSelector": map[string]interface{}{
|
||||
"matchLabels": map[string]interface{}{
|
||||
"kubevirt.io/vm": vmName,
|
||||
},
|
||||
},
|
||||
"endpointSelector": buildEndpointSelector([]string{vmName}),
|
||||
"egress": []interface{}{
|
||||
map[string]interface{}{
|
||||
"toEndpoints": []interface{}{
|
||||
@@ -441,6 +437,13 @@ func (w *WrappedControllerService) addCNPOwnerReference(ctx context.Context, nam
|
||||
if err := unstructured.SetNestedSlice(existing.Object, ownerRefs, "metadata", "ownerReferences"); err != nil {
|
||||
return status.Errorf(codes.Internal, "failed to set ownerReferences: %v", err)
|
||||
}
|
||||
|
||||
// Rebuild endpointSelector to include all VMs
|
||||
selector := buildEndpointSelector(vmNamesFromOwnerRefs(ownerRefs))
|
||||
if err := unstructured.SetNestedField(existing.Object, selector, "spec", "endpointSelector"); err != nil {
|
||||
return status.Errorf(codes.Internal, "failed to set endpointSelector: %v", err)
|
||||
}
|
||||
|
||||
if _, err := w.dynamicClient.Resource(ciliumNetworkPolicyGVR).Namespace(namespace).Update(ctx, existing, metav1.UpdateOptions{}); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -486,6 +489,13 @@ func (w *WrappedControllerService) removeCNPOwnerReference(ctx context.Context,
|
||||
if err := unstructured.SetNestedSlice(existing.Object, remaining, "metadata", "ownerReferences"); err != nil {
|
||||
return status.Errorf(codes.Internal, "failed to set ownerReferences: %v", err)
|
||||
}
|
||||
|
||||
// Rebuild endpointSelector from remaining VMs
|
||||
selector := buildEndpointSelector(vmNamesFromOwnerRefs(remaining))
|
||||
if err := unstructured.SetNestedField(existing.Object, selector, "spec", "endpointSelector"); err != nil {
|
||||
return status.Errorf(codes.Internal, "failed to set endpointSelector: %v", err)
|
||||
}
|
||||
|
||||
if _, err := w.dynamicClient.Resource(ciliumNetworkPolicyGVR).Namespace(namespace).Update(ctx, existing, metav1.UpdateOptions{}); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -494,6 +504,37 @@ func (w *WrappedControllerService) removeCNPOwnerReference(ctx context.Context,
|
||||
})
|
||||
}
|
||||
|
||||
// buildEndpointSelector returns an endpointSelector using matchExpressions
|
||||
// so that multiple VMs can be listed in a single selector.
|
||||
func buildEndpointSelector(vmNames []string) map[string]interface{} {
|
||||
values := make([]interface{}, len(vmNames))
|
||||
for i, name := range vmNames {
|
||||
values[i] = name
|
||||
}
|
||||
return map[string]interface{}{
|
||||
"matchExpressions": []interface{}{
|
||||
map[string]interface{}{
|
||||
"key": "kubevirt.io/vm",
|
||||
"operator": "In",
|
||||
"values": values,
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// vmNamesFromOwnerRefs extracts VM names from ownerReferences.
|
||||
func vmNamesFromOwnerRefs(ownerRefs []interface{}) []string {
|
||||
var names []string
|
||||
for _, ref := range ownerRefs {
|
||||
if refMap, ok := ref.(map[string]interface{}); ok {
|
||||
if name, ok := refMap["name"].(string); ok {
|
||||
names = append(names, name)
|
||||
}
|
||||
}
|
||||
}
|
||||
return names
|
||||
}
|
||||
|
||||
func hasRWXAccessMode(pvc *corev1.PersistentVolumeClaim) bool {
|
||||
for _, mode := range pvc.Spec.AccessModes {
|
||||
if mode == corev1.ReadWriteMany {
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
ttl.sh/rjfkdsjflsk/ubuntu-container-disk:v1.30@sha256:8c2276f68beb67edf5bf76d6c97b271dd9303b336e1d5850ae2b91a590c9bb57
|
||||
@@ -0,0 +1 @@
|
||||
ttl.sh/rjfkdsjflsk/ubuntu-container-disk:v1.31@sha256:2b631cd227bc9b1bae16de033830e756cd6590b512dc0d2b13367ee626f3e4ca
|
||||
@@ -0,0 +1 @@
|
||||
ttl.sh/rjfkdsjflsk/ubuntu-container-disk:v1.32@sha256:600d6ce7df4eaa8cc79c7d6d1b01ecac43e7696beb84eafce752d9210a16455f
|
||||
@@ -0,0 +1 @@
|
||||
ttl.sh/rjfkdsjflsk/ubuntu-container-disk:v1.33@sha256:243e55d6f2887a4f6ce8526de52fd083b7b88194d5c7f3eaa51b87efb557ac88
|
||||
@@ -0,0 +1 @@
|
||||
ttl.sh/rjfkdsjflsk/ubuntu-container-disk:v1.34@sha256:ad8377d5644ba51729dc69dff4c9f6b4a48957075d054a58c61a45d0bb41f6af
|
||||
@@ -0,0 +1 @@
|
||||
ttl.sh/rjfkdsjflsk/ubuntu-container-disk:v1.35@sha256:1c2f2430383a9b9882358c60c194465c1b6092b4aa77536a0343cf74155c0067
|
||||
@@ -1 +0,0 @@
|
||||
ghcr.io/cozystack/cozystack/ubuntu-container-disk:v1.35@sha256:39f626c802dd84f95720ffb54fcd80dfb8a58ac280498870d0a1aa30d4252f94
|
||||
@@ -74,7 +74,7 @@ spec:
|
||||
volumes:
|
||||
- name: system
|
||||
containerDisk:
|
||||
image: "{{ $.Files.Get "images/ubuntu-container-disk.tag" | trim }}"
|
||||
image: "{{ $.Files.Get (printf "images/ubuntu-container-disk-%s.tag" $.Values.version) | trim }}"
|
||||
- name: ephemeral
|
||||
emptyDisk:
|
||||
capacity: {{ .group.ephemeralStorage | default "20Gi" }}
|
||||
@@ -249,6 +249,9 @@ spec:
|
||||
joinConfiguration:
|
||||
nodeRegistration:
|
||||
kubeletExtraArgs: {}
|
||||
# Ignore this for 1.31
|
||||
ignorePreflightErrors:
|
||||
- FileExisting-conntrack
|
||||
discovery:
|
||||
bootstrapToken:
|
||||
apiServerEndpoint: {{ $.Release.Name }}.{{ $.Release.Namespace }}.svc:6443
|
||||
|
||||
@@ -18,5 +18,5 @@ spec:
|
||||
path: system/backupstrategy-controller
|
||||
install:
|
||||
privileged: true
|
||||
namespace: cozy-backupstrategy-controller
|
||||
namespace: cozy-backup-controller
|
||||
releaseName: backupstrategy-controller
|
||||
|
||||
19
packages/core/platform/sources/cozystack-scheduler.yaml
Normal file
19
packages/core/platform/sources/cozystack-scheduler.yaml
Normal file
@@ -0,0 +1,19 @@
|
||||
---
|
||||
apiVersion: cozystack.io/v1alpha1
|
||||
kind: PackageSource
|
||||
metadata:
|
||||
name: cozystack.cozystack-scheduler
|
||||
spec:
|
||||
sourceRef:
|
||||
kind: OCIRepository
|
||||
name: cozystack-packages
|
||||
namespace: cozy-system
|
||||
path: /
|
||||
variants:
|
||||
- name: default
|
||||
components:
|
||||
- name: cozystack-scheduler
|
||||
path: system/cozystack-scheduler
|
||||
install:
|
||||
namespace: kube-system
|
||||
releaseName: cozystack-scheduler
|
||||
@@ -26,6 +26,7 @@ stringData:
|
||||
oidc-enabled: {{ .Values.authentication.oidc.enabled | quote }}
|
||||
oidc-insecure-skip-verify: {{ .Values.authentication.oidc.insecureSkipVerify | quote }}
|
||||
extra-keycloak-redirect-uri-for-dashboard: {{ index .Values.authentication.oidc.keycloakExtraRedirectUri | quote }}
|
||||
keycloak-internal-url: {{ .Values.authentication.oidc.keycloakInternalUrl | quote }}
|
||||
expose-services: {{ .Values.publishing.exposedServices | join "," | quote }}
|
||||
expose-ingress: {{ .Values.publishing.ingressName | quote }}
|
||||
expose-external-ips: {{ .Values.publishing.externalIPs | join "," | quote }}
|
||||
|
||||
@@ -156,5 +156,6 @@
|
||||
{{include "cozystack.platform.package.default" (list "cozystack.bootbox" $) }}
|
||||
{{- end }}
|
||||
{{include "cozystack.platform.package.optional.default" (list "cozystack.hetzner-robotlb" $) }}
|
||||
{{include "cozystack.platform.package.optional.default" (list "cozystack.cozystack-scheduler" $) }}
|
||||
|
||||
{{- end }}
|
||||
|
||||
@@ -54,6 +54,11 @@ authentication:
|
||||
enabled: false
|
||||
insecureSkipVerify: false
|
||||
keycloakExtraRedirectUri: ""
|
||||
# Internal URL to access KeyCloak realm for backend-to-backend requests (bypasses external DNS).
|
||||
# When set, oauth2-proxy uses --skip-oidc-discovery and routes all backend calls (token, jwks,
|
||||
# userinfo, logout) through this URL while keeping browser redirects on the external URL.
|
||||
# Example: http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy
|
||||
keycloakInternalUrl: ""
|
||||
# Pod scheduling configuration
|
||||
scheduling:
|
||||
globalAppTopologySpreadConstraints: ""
|
||||
|
||||
@@ -178,3 +178,7 @@ vmagent:
|
||||
urls:
|
||||
- http://vminsert-shortterm:8480/insert/0/prometheus
|
||||
- http://vminsert-longterm:8480/insert/0/prometheus
|
||||
## inlineScrapeConfig: |
|
||||
## - job_name: "custom"
|
||||
## static_configs:
|
||||
## - targets: ["my-service:9090"]
|
||||
|
||||
@@ -14,3 +14,10 @@ rules:
|
||||
- apiGroups: ["backups.cozystack.io"]
|
||||
resources: ["backupjobs"]
|
||||
verbs: ["create", "get", "list", "watch"]
|
||||
# Leader election (--leader-elect)
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resources: ["leases"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "patch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["events"]
|
||||
verbs: ["create", "patch"]
|
||||
|
||||
@@ -30,6 +30,10 @@ rules:
|
||||
- apiGroups: ["velero.io"]
|
||||
resources: ["backups", "restores"]
|
||||
verbs: ["create", "get", "list", "watch", "update", "patch"]
|
||||
# Events from Recorder.Event() calls
|
||||
- apiGroups: [""]
|
||||
resources: ["events"]
|
||||
verbs: ["create", "patch"]
|
||||
# Leader election (--leader-elect)
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resources: ["leases"]
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
{{- $endpoint := printf "s3.%s" .Values._namespace.host }}
|
||||
{{- range $name, $user := .Values.users }}
|
||||
{{- $secretName := printf "%s-%s" $.Values.bucketName $name }}
|
||||
{{- $existingSecret := lookup "v1" "Secret" $.Release.Namespace $secretName }}
|
||||
{{- if $existingSecret }}
|
||||
{{- $bucketInfo := fromJson (b64dec (index $existingSecret.data "BucketInfo")) }}
|
||||
{{- $endpoint = trimPrefix "https://" (index $bucketInfo.spec.secretS3 "endpoint") }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
@@ -17,6 +26,6 @@ spec:
|
||||
image: "{{ $.Files.Get "images/s3manager.tag" | trim }}"
|
||||
env:
|
||||
- name: ENDPOINT
|
||||
value: "s3.{{ .Values._namespace.host }}"
|
||||
value: {{ $endpoint | quote }}
|
||||
- name: SKIP_SSL_VERIFICATION
|
||||
value: "true"
|
||||
|
||||
3
packages/system/cozystack-scheduler/Chart.yaml
Normal file
3
packages/system/cozystack-scheduler/Chart.yaml
Normal file
@@ -0,0 +1,3 @@
|
||||
apiVersion: v2
|
||||
name: cozy-cozystack-scheduler
|
||||
version: 0.1.0
|
||||
10
packages/system/cozystack-scheduler/Makefile
Normal file
10
packages/system/cozystack-scheduler/Makefile
Normal file
@@ -0,0 +1,10 @@
|
||||
export NAME=cozystack-scheduler
|
||||
export NAMESPACE=kube-system
|
||||
|
||||
include ../../../hack/package.mk
|
||||
|
||||
update:
|
||||
rm -rf crds templates values.yaml Chart.yaml
|
||||
tag=$$(git ls-remote --tags --sort="v:refname" https://github.com/cozystack/cozystack-scheduler | awk -F'[/^]' 'END{print $$3}') && \
|
||||
curl -sSL https://github.com/cozystack/cozystack-scheduler/archive/refs/tags/$${tag}.tar.gz | \
|
||||
tar xzvf - --strip 2 cozystack-scheduler-$${tag#*v}/chart
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: cozystack-scheduler
|
||||
rules:
|
||||
- apiGroups: ["cozystack.io"]
|
||||
resources:
|
||||
- schedulingclasses
|
||||
verbs: ["get", "list", "watch"]
|
||||
@@ -0,0 +1,38 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: cozystack-scheduler:kube-scheduler
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: system:kube-scheduler
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cozystack-scheduler
|
||||
namespace: {{ .Release.Namespace }}
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: cozystack-scheduler:volume-scheduler
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: system:volume-scheduler
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cozystack-scheduler
|
||||
namespace: {{ .Release.Namespace }}
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: cozystack-scheduler
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: cozystack-scheduler
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cozystack-scheduler
|
||||
namespace: {{ .Release.Namespace }}
|
||||
54
packages/system/cozystack-scheduler/templates/configmap.yaml
Normal file
54
packages/system/cozystack-scheduler/templates/configmap.yaml
Normal file
@@ -0,0 +1,54 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: cozystack-scheduler-config
|
||||
namespace: {{ .Release.Namespace }}
|
||||
data:
|
||||
scheduler-config.yaml: |
|
||||
apiVersion: kubescheduler.config.k8s.io/v1
|
||||
kind: KubeSchedulerConfiguration
|
||||
leaderElection:
|
||||
leaderElect: true
|
||||
resourceNamespace: {{ .Release.Namespace }}
|
||||
resourceName: cozystack-scheduler
|
||||
profiles:
|
||||
- schedulerName: cozystack-scheduler
|
||||
plugins:
|
||||
preFilter:
|
||||
disabled:
|
||||
- name: InterPodAffinity
|
||||
- name: NodeAffinity
|
||||
- name: PodTopologySpread
|
||||
enabled:
|
||||
- name: CozystackInterPodAffinity
|
||||
- name: CozystackNodeAffinity
|
||||
- name: CozystackPodTopologySpread
|
||||
- name: CozystackSchedulingClass
|
||||
filter:
|
||||
disabled:
|
||||
- name: InterPodAffinity
|
||||
- name: NodeAffinity
|
||||
- name: PodTopologySpread
|
||||
enabled:
|
||||
- name: CozystackInterPodAffinity
|
||||
- name: CozystackNodeAffinity
|
||||
- name: CozystackPodTopologySpread
|
||||
- name: CozystackSchedulingClass
|
||||
preScore:
|
||||
disabled:
|
||||
- name: InterPodAffinity
|
||||
- name: NodeAffinity
|
||||
- name: PodTopologySpread
|
||||
enabled:
|
||||
- name: CozystackInterPodAffinity
|
||||
- name: CozystackNodeAffinity
|
||||
- name: CozystackPodTopologySpread
|
||||
score:
|
||||
disabled:
|
||||
- name: InterPodAffinity
|
||||
- name: NodeAffinity
|
||||
- name: PodTopologySpread
|
||||
enabled:
|
||||
- name: CozystackInterPodAffinity
|
||||
- name: CozystackNodeAffinity
|
||||
- name: CozystackPodTopologySpread
|
||||
@@ -0,0 +1,37 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cozystack-scheduler
|
||||
namespace: {{ .Release.Namespace }}
|
||||
spec:
|
||||
replicas: {{ .Values.replicas }}
|
||||
selector:
|
||||
matchLabels:
|
||||
app: cozystack-scheduler
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: cozystack-scheduler
|
||||
spec:
|
||||
serviceAccountName: cozystack-scheduler
|
||||
containers:
|
||||
- name: cozystack-scheduler
|
||||
image: {{ .Values.image }}
|
||||
command:
|
||||
- /cozystack-scheduler
|
||||
- --config=/etc/kubernetes/scheduler-config.yaml
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: 10259
|
||||
scheme: HTTPS
|
||||
initialDelaySeconds: 15
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/kubernetes/scheduler-config.yaml
|
||||
subPath: scheduler-config.yaml
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: cozystack-scheduler-config
|
||||
@@ -0,0 +1,40 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: cozystack-scheduler:extension-apiserver-authentication-reader
|
||||
namespace: kube-system
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: extension-apiserver-authentication-reader
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cozystack-scheduler
|
||||
namespace: {{ .Release.Namespace }}
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: cozystack-scheduler:leader-election
|
||||
namespace: {{ .Release.Namespace }}
|
||||
rules:
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resources: ["leases"]
|
||||
verbs: ["create", "get", "list", "update", "watch"]
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resources: ["leasecandidates"]
|
||||
verbs: ["create", "get", "list", "update", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: cozystack-scheduler:leader-election
|
||||
namespace: {{ .Release.Namespace }}
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: cozystack-scheduler:leader-election
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: cozystack-scheduler
|
||||
namespace: {{ .Release.Namespace }}
|
||||
@@ -0,0 +1,5 @@
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: cozystack-scheduler
|
||||
namespace: {{ .Release.Namespace }}
|
||||
2
packages/system/cozystack-scheduler/values.yaml
Normal file
2
packages/system/cozystack-scheduler/values.yaml
Normal file
@@ -0,0 +1,2 @@
|
||||
image: ghcr.io/cozystack/cozystack/cozystack-scheduler:v0.1.0@sha256:5f7150c82177478467ff80628acb5a400291aff503364aa9e26fc346d79a73cf
|
||||
replicas: 1
|
||||
@@ -1,6 +1,7 @@
|
||||
{{- $host := index .Values._cluster "root-host" }}
|
||||
{{- $oidcEnabled := index .Values._cluster "oidc-enabled" }}
|
||||
{{- $oidcInsecureSkipVerify := index .Values._cluster "oidc-insecure-skip-verify" }}
|
||||
{{- $keycloakInternalUrl := index .Values._cluster "keycloak-internal-url" | default "" }}
|
||||
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -55,7 +56,16 @@ spec:
|
||||
- --http-address=0.0.0.0:8000
|
||||
- --redirect-url=https://dashboard.{{ $host }}/oauth2/callback
|
||||
- --oidc-issuer-url=https://keycloak.{{ $host }}/realms/cozy
|
||||
{{- if $keycloakInternalUrl }}
|
||||
- --skip-oidc-discovery
|
||||
- --login-url=https://keycloak.{{ $host }}/realms/cozy/protocol/openid-connect/auth
|
||||
- --redeem-url={{ $keycloakInternalUrl }}/protocol/openid-connect/token
|
||||
- --oidc-jwks-url={{ $keycloakInternalUrl }}/protocol/openid-connect/certs
|
||||
- --validate-url={{ $keycloakInternalUrl }}/protocol/openid-connect/userinfo
|
||||
- --backend-logout-url={{ $keycloakInternalUrl }}/protocol/openid-connect/logout?id_token_hint={id_token}
|
||||
{{- else }}
|
||||
- --backend-logout-url=https://keycloak.{{ $host }}/realms/cozy/protocol/openid-connect/logout?id_token_hint={id_token}
|
||||
{{- end }}
|
||||
- --whitelist-domain=keycloak.{{ $host }}
|
||||
- --email-domain=*
|
||||
- --pass-access-token=true
|
||||
|
||||
@@ -3,7 +3,7 @@ kind: CDI
|
||||
metadata:
|
||||
name: cdi
|
||||
spec:
|
||||
cloneStrategyOverride: copy
|
||||
cloneStrategyOverride: csi-clone
|
||||
config:
|
||||
{{- with .Values.uploadProxyURL }}
|
||||
uploadProxyURLOverride: {{ quote . }}
|
||||
|
||||
@@ -0,0 +1,358 @@
|
||||
diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go
|
||||
index f544493..98e7fde 100644
|
||||
--- a/pkg/client/linstor.go
|
||||
+++ b/pkg/client/linstor.go
|
||||
@@ -181,7 +181,7 @@ func LogLevel(s string) func(*Linstor) error {
|
||||
func (s *Linstor) ListAllWithStatus(ctx context.Context) ([]volume.VolumeStatus, error) {
|
||||
var vols []volume.VolumeStatus
|
||||
|
||||
- resourcesByName := make(map[string][]lapi.Resource)
|
||||
+ resourcesByName := make(map[string][]lapi.ResourceWithVolumes)
|
||||
|
||||
resDefs, err := s.client.ResourceDefinitions.GetAll(ctx, lapi.RDGetAllRequest{WithVolumeDefinitions: true})
|
||||
if err != nil {
|
||||
@@ -194,7 +194,7 @@ func (s *Linstor) ListAllWithStatus(ctx context.Context) ([]volume.VolumeStatus,
|
||||
}
|
||||
|
||||
for i := range allResources {
|
||||
- resourcesByName[allResources[i].Name] = append(resourcesByName[allResources[i].Name], allResources[i].Resource)
|
||||
+ resourcesByName[allResources[i].Name] = append(resourcesByName[allResources[i].Name], allResources[i])
|
||||
}
|
||||
|
||||
for _, rd := range resDefs {
|
||||
@@ -462,6 +462,14 @@ func (s *Linstor) Clone(ctx context.Context, vol, src *volume.Info, params *volu
|
||||
return err
|
||||
}
|
||||
|
||||
+ if params.RelocateAfterClone {
|
||||
+ logger.Debug("relocate resources to optimal nodes")
|
||||
+
|
||||
+ if err := s.relocateResources(ctx, vol.ID, rGroup.Name); err != nil {
|
||||
+ logger.WithError(err).Warn("resource relocation failed, volume is still usable")
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
logger.Debug("reconcile extra properties")
|
||||
|
||||
err = s.client.ResourceDefinitions.Modify(ctx, vol.ID, lapi.GenericPropsModify{OverrideProps: vol.Properties})
|
||||
@@ -1280,6 +1288,14 @@ func (s *Linstor) VolFromSnap(ctx context.Context, snap *volume.Snapshot, vol *v
|
||||
return err
|
||||
}
|
||||
|
||||
+ if snapParams != nil && snapParams.RelocateAfterRestore {
|
||||
+ logger.Debug("relocate resources to optimal nodes")
|
||||
+
|
||||
+ if err := s.relocateResources(ctx, vol.ID, rGroup.Name); err != nil {
|
||||
+ logger.WithError(err).Warn("resource relocation failed, volume is still usable")
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
logger.Debug("reconcile extra properties")
|
||||
|
||||
err = s.client.ResourceDefinitions.Modify(ctx, vol.ID, lapi.GenericPropsModify{OverrideProps: vol.Properties})
|
||||
@@ -1470,9 +1486,8 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu
|
||||
return fmt.Errorf("snapshot '%s' not deployed on any node", snap.Name)
|
||||
}
|
||||
|
||||
- // Optimize the node we use to restore. It should be one of the preferred nodes, or just the first with a snapshot
|
||||
- // if no preferred nodes match.
|
||||
- var selectedNode string
|
||||
+ // Collect available snapshot nodes, preferring those matching the topology hints.
|
||||
+ var preferred, available []string
|
||||
for _, snapNode := range snap.Nodes {
|
||||
if err := s.NodeAvailable(ctx, snapNode); err != nil {
|
||||
logger.WithField("selected node candidate", snapNode).WithError(err).Debug("node is not available")
|
||||
@@ -1480,13 +1495,23 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu
|
||||
}
|
||||
|
||||
if slices.Contains(preferredNodes, snapNode) {
|
||||
- // We found a perfect candidate.
|
||||
- selectedNode = snapNode
|
||||
- break
|
||||
- } else if selectedNode == "" {
|
||||
- // Set a fallback if we have no candidate yet.
|
||||
- selectedNode = snapNode
|
||||
+ preferred = append(preferred, snapNode)
|
||||
}
|
||||
+
|
||||
+ available = append(available, snapNode)
|
||||
+ }
|
||||
+
|
||||
+ // Pick a random node from preferred candidates, falling back to any available node.
|
||||
+ // Randomization distributes restore load across snapshot nodes, preventing all
|
||||
+ // clones of the same source from concentrating on a single node.
|
||||
+ candidates := preferred
|
||||
+ if len(candidates) == 0 {
|
||||
+ candidates = available
|
||||
+ }
|
||||
+
|
||||
+ var selectedNode string
|
||||
+ if len(candidates) > 0 {
|
||||
+ selectedNode = candidates[rand.Intn(len(candidates))]
|
||||
}
|
||||
|
||||
if selectedNode == "" {
|
||||
@@ -1679,6 +1704,114 @@ func (s *Linstor) reconcileResourcePlacement(ctx context.Context, vol *volume.In
|
||||
return nil
|
||||
}
|
||||
|
||||
+const propertyRelocationTriggered = "Aux/csi-relocation-triggered"
|
||||
+
|
||||
+// relocateResources migrates replicas from their current nodes to the nodes that
|
||||
+// LINSTOR's autoplacer considers optimal. This is a best-effort, fire-and-forget
|
||||
+// operation: migrate-disk API is asynchronous and the volume remains usable on its
|
||||
+// current nodes even if relocation fails.
|
||||
+func (s *Linstor) relocateResources(ctx context.Context, volID, rgName string) error {
|
||||
+ logger := s.log.WithFields(logrus.Fields{
|
||||
+ "volume": volID,
|
||||
+ "resourceGroup": rgName,
|
||||
+ })
|
||||
+
|
||||
+ // Check if relocation was already triggered (idempotency guard).
|
||||
+ rd, err := s.client.ResourceDefinitions.Get(ctx, volID)
|
||||
+ if err != nil {
|
||||
+ return fmt.Errorf("failed to get resource definition: %w", err)
|
||||
+ }
|
||||
+
|
||||
+ if rd.Props[propertyRelocationTriggered] != "" {
|
||||
+ logger.Debug("relocation already triggered, skipping")
|
||||
+ return nil
|
||||
+ }
|
||||
+
|
||||
+ // Get current diskful nodes.
|
||||
+ resources, err := s.client.Resources.GetAll(ctx, volID)
|
||||
+ if err != nil {
|
||||
+ return fmt.Errorf("failed to list resources: %w", err)
|
||||
+ }
|
||||
+
|
||||
+ currentNodes := util.DeployedDiskfullyNodes(resources)
|
||||
+
|
||||
+ // Query optimal placement from LINSTOR autoplacer.
|
||||
+ sizeInfo, err := s.client.ResourceGroups.QuerySizeInfo(ctx, rgName, lapi.QuerySizeInfoRequest{})
|
||||
+ if err != nil {
|
||||
+ return fmt.Errorf("failed to query size info: %w", err)
|
||||
+ }
|
||||
+
|
||||
+ if sizeInfo.SpaceInfo == nil || len(sizeInfo.SpaceInfo.NextSpawnResult) == 0 {
|
||||
+ logger.Debug("no spawn result from query-size-info, skipping relocation")
|
||||
+ return nil
|
||||
+ }
|
||||
+
|
||||
+ // Build a map of optimal node -> storage pool.
|
||||
+ optimalPools := make(map[string]string, len(sizeInfo.SpaceInfo.NextSpawnResult))
|
||||
+ for _, r := range sizeInfo.SpaceInfo.NextSpawnResult {
|
||||
+ optimalPools[r.NodeName] = r.StorPoolName
|
||||
+ }
|
||||
+
|
||||
+ // Compute nodes to remove (current but not optimal) and nodes to add (optimal but not current).
|
||||
+ var nodesToRemove, nodesToAdd []string
|
||||
+
|
||||
+ for _, node := range currentNodes {
|
||||
+ if _, ok := optimalPools[node]; !ok {
|
||||
+ nodesToRemove = append(nodesToRemove, node)
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
+ for _, r := range sizeInfo.SpaceInfo.NextSpawnResult {
|
||||
+ if !slices.Contains(currentNodes, r.NodeName) {
|
||||
+ nodesToAdd = append(nodesToAdd, r.NodeName)
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
+ // Only migrate min(remove, add) pairs.
|
||||
+ pairs := min(len(nodesToRemove), len(nodesToAdd))
|
||||
+ if pairs == 0 {
|
||||
+ logger.Debug("no relocation needed, placement is already optimal")
|
||||
+ return nil
|
||||
+ }
|
||||
+
|
||||
+ logger.Infof("relocating %d replica(s): %v -> %v", pairs, nodesToRemove[:pairs], nodesToAdd[:pairs])
|
||||
+
|
||||
+ // Mark relocation as triggered before starting migrations.
|
||||
+ err = s.client.ResourceDefinitions.Modify(ctx, volID, lapi.GenericPropsModify{
|
||||
+ OverrideProps: map[string]string{propertyRelocationTriggered: "true"},
|
||||
+ })
|
||||
+ if err != nil {
|
||||
+ return fmt.Errorf("failed to set relocation property: %w", err)
|
||||
+ }
|
||||
+
|
||||
+ // Migrate each pair sequentially (LINSTOR serializes at the RD level anyway).
|
||||
+ for i := range pairs {
|
||||
+ fromNode := nodesToRemove[i]
|
||||
+ toNode := nodesToAdd[i]
|
||||
+ storPool := optimalPools[toNode]
|
||||
+
|
||||
+ logger := logger.WithFields(logrus.Fields{"from": fromNode, "to": toNode, "storagePool": storPool})
|
||||
+
|
||||
+ logger.Info("creating diskless resource on target node")
|
||||
+
|
||||
+ err := s.client.Resources.MakeAvailable(ctx, volID, toNode, lapi.ResourceMakeAvailable{})
|
||||
+ if err != nil {
|
||||
+ logger.WithError(err).Warn("failed to create diskless on target, skipping this pair")
|
||||
+ continue
|
||||
+ }
|
||||
+
|
||||
+ logger.Info("initiating migrate-disk")
|
||||
+
|
||||
+ err = s.client.Resources.Migrate(ctx, volID, fromNode, toNode, storPool)
|
||||
+ if err != nil {
|
||||
+ logger.WithError(err).Warn("migrate-disk failed, skipping this pair")
|
||||
+ continue
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
+ return nil
|
||||
+}
|
||||
+
|
||||
// FindSnapsByID searches the snapshot in the backend
|
||||
func (s *Linstor) FindSnapsByID(ctx context.Context, id string) ([]*volume.Snapshot, error) {
|
||||
snapshotId, err := volume.ParseSnapshotId(id)
|
||||
@@ -2173,7 +2306,7 @@ func (s *Linstor) Status(ctx context.Context, volId string) ([]string, *csi.Volu
|
||||
"volume": volId,
|
||||
}).Debug("getting assignments")
|
||||
|
||||
- ress, err := s.client.Resources.GetAll(ctx, volId)
|
||||
+ ress, err := s.client.Resources.GetResourceView(ctx, &lapi.ListOpts{Resource: []string{volId}})
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("failed to list resources for '%s': %w", volId, err)
|
||||
}
|
||||
@@ -2261,13 +2394,20 @@ func GetSnapshotRemoteAndReadiness(snap *lapi.Snapshot) (string, bool, error) {
|
||||
return "", slices.Contains(snap.Flags, lapiconsts.FlagSuccessful), nil
|
||||
}
|
||||
|
||||
-func NodesAndConditionFromResources(ress []lapi.Resource) ([]string, *csi.VolumeCondition) {
|
||||
+func NodesAndConditionFromResources(ress []lapi.ResourceWithVolumes) ([]string, *csi.VolumeCondition) {
|
||||
var allNodes, abnormalNodes []string
|
||||
|
||||
for i := range ress {
|
||||
res := &ress[i]
|
||||
|
||||
- allNodes = append(allNodes, res.NodeName)
|
||||
+ // A resource is a CSI publish target if any of its volumes were created
|
||||
+ // by ControllerPublishVolume, identified by the temporary-diskless-attach property.
|
||||
+ if slices.ContainsFunc(res.Volumes, func(v lapi.Volume) bool {
|
||||
+ createdFor, ok := v.Props[linstor.PropertyCreatedFor]
|
||||
+ return ok && createdFor == linstor.CreatedForTemporaryDisklessAttach
|
||||
+ }) {
|
||||
+ allNodes = append(allNodes, res.NodeName)
|
||||
+ }
|
||||
|
||||
if res.State == nil {
|
||||
abnormalNodes = append(abnormalNodes, res.NodeName)
|
||||
diff --git a/pkg/volume/parameter.go b/pkg/volume/parameter.go
|
||||
index 39acd95..aed18ab 100644
|
||||
--- a/pkg/volume/parameter.go
|
||||
+++ b/pkg/volume/parameter.go
|
||||
@@ -50,6 +50,7 @@ const (
|
||||
nfsservicename
|
||||
nfssquash
|
||||
nfsrecoveryvolumesize
|
||||
+ relocateafterclone
|
||||
)
|
||||
|
||||
// Parameters configuration for linstor volumes.
|
||||
@@ -118,6 +119,8 @@ type Parameters struct {
|
||||
// NfsRecoveryVolumeSize sets the volume size (in bytes) of the recovery volume used by the NFS server.
|
||||
// Defaults to 300MiB.
|
||||
NfsRecoveryVolumeBytes int64
|
||||
+ // RelocateAfterClone triggers asynchronous relocation of replicas to optimal nodes after cloning.
|
||||
+ RelocateAfterClone bool
|
||||
}
|
||||
|
||||
const DefaultDisklessStoragePoolName = "DfltDisklessStorPool"
|
||||
@@ -140,6 +143,7 @@ func NewParameters(params map[string]string, topologyPrefix string) (Parameters,
|
||||
NfsServiceName: "linstor-csi-nfs",
|
||||
NfsSquash: "no_root_squash",
|
||||
NfsRecoveryVolumeBytes: 300 * 1024 * 1024,
|
||||
+ RelocateAfterClone: true,
|
||||
}
|
||||
|
||||
for k, v := range params {
|
||||
@@ -287,6 +291,13 @@ func NewParameters(params map[string]string, topologyPrefix string) (Parameters,
|
||||
}
|
||||
|
||||
p.NfsRecoveryVolumeBytes = s.Value()
|
||||
+ case relocateafterclone:
|
||||
+ val, err := strconv.ParseBool(v)
|
||||
+ if err != nil {
|
||||
+ return p, err
|
||||
+ }
|
||||
+
|
||||
+ p.RelocateAfterClone = val
|
||||
}
|
||||
}
|
||||
|
||||
diff --git a/pkg/volume/paramkey_enumer.go b/pkg/volume/paramkey_enumer.go
|
||||
index 5474963..2eb8a7c 100644
|
||||
--- a/pkg/volume/paramkey_enumer.go
|
||||
+++ b/pkg/volume/paramkey_enumer.go
|
||||
@@ -7,11 +7,11 @@ import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
-const _paramKeyName = "allowremotevolumeaccessautoplaceclientlistdisklessonremainingdisklessstoragepooldonotplacewithregexencryptionfsoptslayerlistmountoptsnodelistplacementcountplacementpolicyreplicasondifferentreplicasonsamesizekibstoragepoolpostmountxfsoptsresourcegroupusepvcnameoverprovisionxreplicasondifferentnfsconfigtemplatenfsservicenamenfssquashnfsrecoveryvolumesize"
|
||||
+const _paramKeyName = "allowremotevolumeaccessautoplaceclientlistdisklessonremainingdisklessstoragepooldonotplacewithregexencryptionfsoptslayerlistmountoptsnodelistplacementcountplacementpolicyreplicasondifferentreplicasonsamesizekibstoragepoolpostmountxfsoptsresourcegroupusepvcnameoverprovisionxreplicasondifferentnfsconfigtemplatenfsservicenamenfssquashnfsrecoveryvolumesizerelocateafterclone"
|
||||
|
||||
-var _paramKeyIndex = [...]uint16{0, 23, 32, 42, 61, 80, 99, 109, 115, 124, 133, 141, 155, 170, 189, 203, 210, 221, 237, 250, 260, 273, 293, 310, 324, 333, 354}
|
||||
+var _paramKeyIndex = [...]uint16{0, 23, 32, 42, 61, 80, 99, 109, 115, 124, 133, 141, 155, 170, 189, 203, 210, 221, 237, 250, 260, 273, 293, 310, 324, 333, 354, 372}
|
||||
|
||||
-const _paramKeyLowerName = "allowremotevolumeaccessautoplaceclientlistdisklessonremainingdisklessstoragepooldonotplacewithregexencryptionfsoptslayerlistmountoptsnodelistplacementcountplacementpolicyreplicasondifferentreplicasonsamesizekibstoragepoolpostmountxfsoptsresourcegroupusepvcnameoverprovisionxreplicasondifferentnfsconfigtemplatenfsservicenamenfssquashnfsrecoveryvolumesize"
|
||||
+const _paramKeyLowerName = "allowremotevolumeaccessautoplaceclientlistdisklessonremainingdisklessstoragepooldonotplacewithregexencryptionfsoptslayerlistmountoptsnodelistplacementcountplacementpolicyreplicasondifferentreplicasonsamesizekibstoragepoolpostmountxfsoptsresourcegroupusepvcnameoverprovisionxreplicasondifferentnfsconfigtemplatenfsservicenamenfssquashnfsrecoveryvolumesizerelocateafterclone"
|
||||
|
||||
func (i paramKey) String() string {
|
||||
if i < 0 || i >= paramKey(len(_paramKeyIndex)-1) {
|
||||
@@ -50,9 +50,10 @@ func _paramKeyNoOp() {
|
||||
_ = x[nfsservicename-(23)]
|
||||
_ = x[nfssquash-(24)]
|
||||
_ = x[nfsrecoveryvolumesize-(25)]
|
||||
+ _ = x[relocateafterclone-(26)]
|
||||
}
|
||||
|
||||
-var _paramKeyValues = []paramKey{allowremotevolumeaccess, autoplace, clientlist, disklessonremaining, disklessstoragepool, donotplacewithregex, encryption, fsopts, layerlist, mountopts, nodelist, placementcount, placementpolicy, replicasondifferent, replicasonsame, sizekib, storagepool, postmountxfsopts, resourcegroup, usepvcname, overprovision, xreplicasondifferent, nfsconfigtemplate, nfsservicename, nfssquash, nfsrecoveryvolumesize}
|
||||
+var _paramKeyValues = []paramKey{allowremotevolumeaccess, autoplace, clientlist, disklessonremaining, disklessstoragepool, donotplacewithregex, encryption, fsopts, layerlist, mountopts, nodelist, placementcount, placementpolicy, replicasondifferent, replicasonsame, sizekib, storagepool, postmountxfsopts, resourcegroup, usepvcname, overprovision, xreplicasondifferent, nfsconfigtemplate, nfsservicename, nfssquash, nfsrecoveryvolumesize, relocateafterclone}
|
||||
|
||||
var _paramKeyNameToValueMap = map[string]paramKey{
|
||||
_paramKeyName[0:23]: allowremotevolumeaccess,
|
||||
@@ -107,6 +108,8 @@ var _paramKeyNameToValueMap = map[string]paramKey{
|
||||
_paramKeyLowerName[324:333]: nfssquash,
|
||||
_paramKeyName[333:354]: nfsrecoveryvolumesize,
|
||||
_paramKeyLowerName[333:354]: nfsrecoveryvolumesize,
|
||||
+ _paramKeyName[354:372]: relocateafterclone,
|
||||
+ _paramKeyLowerName[354:372]: relocateafterclone,
|
||||
}
|
||||
|
||||
var _paramKeyNames = []string{
|
||||
@@ -136,6 +139,7 @@ var _paramKeyNames = []string{
|
||||
_paramKeyName[310:324],
|
||||
_paramKeyName[324:333],
|
||||
_paramKeyName[333:354],
|
||||
+ _paramKeyName[354:372],
|
||||
}
|
||||
|
||||
// paramKeyString retrieves an enum value from the enum constants string name.
|
||||
diff --git a/pkg/volume/snapshot_params.go b/pkg/volume/snapshot_params.go
|
||||
index d167cb8..50b70fb 100644
|
||||
--- a/pkg/volume/snapshot_params.go
|
||||
+++ b/pkg/volume/snapshot_params.go
|
||||
@@ -35,6 +35,7 @@ type SnapshotParameters struct {
|
||||
LinstorTargetUrl string `json:"linstor-target-url,omitempty"`
|
||||
LinstorTargetClusterID string `json:"linstor-target-cluster-id,omitempty"`
|
||||
LinstorTargetStoragePool string `json:"linstor-target-storage-pool,omitempty"`
|
||||
+ RelocateAfterRestore bool `json:"relocate-after-restore,omitempty"`
|
||||
}
|
||||
|
||||
func NewSnapshotParameters(params, secrets map[string]string) (*SnapshotParameters, error) {
|
||||
@@ -91,6 +92,13 @@ func NewSnapshotParameters(params, secrets map[string]string) (*SnapshotParamete
|
||||
p.LinstorTargetClusterID = v
|
||||
case "/linstor-target-storage-pool":
|
||||
p.LinstorTargetStoragePool = v
|
||||
+ case "/relocateAfterRestore":
|
||||
+ b, err := strconv.ParseBool(v)
|
||||
+ if err != nil {
|
||||
+ return nil, err
|
||||
+ }
|
||||
+
|
||||
+ p.RelocateAfterRestore = b
|
||||
default:
|
||||
log.WithField("key", k).Warn("ignoring unknown snapshot parameter key")
|
||||
}
|
||||
@@ -1,718 +0,0 @@
|
||||
diff --git a/cmd/linstor-csi/linstor-csi.go b/cmd/linstor-csi/linstor-csi.go
|
||||
index 143f6cee..bd28e06e 100644
|
||||
--- a/cmd/linstor-csi/linstor-csi.go
|
||||
+++ b/cmd/linstor-csi/linstor-csi.go
|
||||
@@ -41,22 +41,23 @@ import (
|
||||
|
||||
func main() {
|
||||
var (
|
||||
- lsEndpoint = flag.String("linstor-endpoint", "", "Controller API endpoint for LINSTOR")
|
||||
- lsSkipTLSVerification = flag.Bool("linstor-skip-tls-verification", false, "If true, do not verify tls")
|
||||
- csiEndpoint = flag.String("csi-endpoint", "unix:///var/lib/kubelet/plugins/linstor.csi.linbit.com/csi.sock", "CSI endpoint")
|
||||
- node = flag.String("node", "", "Node ID to pass to node service")
|
||||
- logLevel = flag.String("log-level", "info", "Enable debug log output. Choose from: panic, fatal, error, warn, info, debug")
|
||||
- rps = flag.Float64("linstor-api-requests-per-second", 0, "Maximum allowed number of LINSTOR API requests per second. Default: Unlimited")
|
||||
- burst = flag.Int("linstor-api-burst", 1, "Maximum number of API requests allowed before being limited by requests-per-second. Default: 1 (no bursting)")
|
||||
- bearerTokenFile = flag.String("bearer-token", "", "Read the bearer token from the given file and use it for authentication.")
|
||||
- propNs = flag.String("property-namespace", linstor.NamespcAuxiliary, "Limit the reported topology keys to properties from the given namespace.")
|
||||
- labelBySP = flag.Bool("label-by-storage-pool", true, "Set to false to disable labeling of nodes based on their configured storage pools.")
|
||||
- nodeCacheTimeout = flag.Duration("node-cache-timeout", 1*time.Minute, "Duration for which the results of node and storage pool related API responses should be cached.")
|
||||
- resourceCacheTimeout = flag.Duration("resource-cache-timeout", 30*time.Second, "Duration for which the results of resource related API responses should be cached.")
|
||||
- resyncAfter = flag.Duration("resync-after", 5*time.Minute, "Duration after which reconciliations (such as for VolumeSnapshotClasses) should be rerun. Set to 0 to disable.")
|
||||
- enableRWX = flag.Bool("enable-rwx", false, "Enable RWX support via NFS (requires running in Kubernetes).")
|
||||
- namespace = flag.String("nfs-service-namespace", "", "The namespace the NFS service is running in.")
|
||||
- reactorConfigMapName = flag.String("nfs-reactor-config-map-name", "linstor-csi-nfs-reactor-config", "Name of the config map used to store promoter configuration")
|
||||
+ lsEndpoint = flag.String("linstor-endpoint", "", "Controller API endpoint for LINSTOR")
|
||||
+ lsSkipTLSVerification = flag.Bool("linstor-skip-tls-verification", false, "If true, do not verify tls")
|
||||
+ csiEndpoint = flag.String("csi-endpoint", "unix:///var/lib/kubelet/plugins/linstor.csi.linbit.com/csi.sock", "CSI endpoint")
|
||||
+ node = flag.String("node", "", "Node ID to pass to node service")
|
||||
+ logLevel = flag.String("log-level", "info", "Enable debug log output. Choose from: panic, fatal, error, warn, info, debug")
|
||||
+ rps = flag.Float64("linstor-api-requests-per-second", 0, "Maximum allowed number of LINSTOR API requests per second. Default: Unlimited")
|
||||
+ burst = flag.Int("linstor-api-burst", 1, "Maximum number of API requests allowed before being limited by requests-per-second. Default: 1 (no bursting)")
|
||||
+ bearerTokenFile = flag.String("bearer-token", "", "Read the bearer token from the given file and use it for authentication.")
|
||||
+ propNs = flag.String("property-namespace", linstor.NamespcAuxiliary, "Limit the reported topology keys to properties from the given namespace.")
|
||||
+ labelBySP = flag.Bool("label-by-storage-pool", true, "Set to false to disable labeling of nodes based on their configured storage pools.")
|
||||
+ nodeCacheTimeout = flag.Duration("node-cache-timeout", 1*time.Minute, "Duration for which the results of node and storage pool related API responses should be cached.")
|
||||
+ resourceCacheTimeout = flag.Duration("resource-cache-timeout", 30*time.Second, "Duration for which the results of resource related API responses should be cached.")
|
||||
+ resyncAfter = flag.Duration("resync-after", 5*time.Minute, "Duration after which reconciliations (such as for VolumeSnapshotClasses) should be rerun. Set to 0 to disable.")
|
||||
+ enableRWX = flag.Bool("enable-rwx", false, "Enable RWX support via NFS (requires running in Kubernetes).")
|
||||
+ namespace = flag.String("nfs-service-namespace", "", "The namespace the NFS service is running in.")
|
||||
+ reactorConfigMapName = flag.String("nfs-reactor-config-map-name", "linstor-csi-nfs-reactor-config", "Name of the config map used to store promoter configuration")
|
||||
+ disableRWXBlockValidation = flag.Bool("disable-rwx-block-validation", false, "Disable KubeVirt VM ownership validation for RWX block volumes.")
|
||||
)
|
||||
|
||||
flag.Var(&volume.DefaultRemoteAccessPolicy, "default-remote-access-policy", "")
|
||||
@@ -169,6 +170,10 @@ func main() {
|
||||
opts = append(opts, driver.ConfigureRWX(*namespace, *reactorConfigMapName))
|
||||
}
|
||||
|
||||
+ if *disableRWXBlockValidation {
|
||||
+ opts = append(opts, driver.DisableRWXBlockValidation())
|
||||
+ }
|
||||
+
|
||||
drv, err := driver.NewDriver(opts...)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
diff --git a/pkg/driver/driver.go b/pkg/driver/driver.go
|
||||
index bea69a8b..a39674b6 100644
|
||||
--- a/pkg/driver/driver.go
|
||||
+++ b/pkg/driver/driver.go
|
||||
@@ -83,6 +83,8 @@ type Driver struct {
|
||||
topologyPrefix string
|
||||
// resyncAfter is the interval after which reconciliations should be retried
|
||||
resyncAfter time.Duration
|
||||
+ // disableRWXBlockValidation disables KubeVirt VM ownership validation for RWX block volumes
|
||||
+ disableRWXBlockValidation bool
|
||||
|
||||
// Embed for forward compatibility.
|
||||
csi.UnimplementedIdentityServer
|
||||
@@ -300,6 +302,17 @@ func ResyncAfter(resyncAfter time.Duration) func(*Driver) error {
|
||||
}
|
||||
}
|
||||
|
||||
+// DisableRWXBlockValidation disables the KubeVirt VM ownership validation for RWX block volumes.
|
||||
+// When disabled, the driver will not check if multiple pods using the same RWX block volume
|
||||
+// belong to the same VM. This may be needed in environments where the validation causes issues
|
||||
+// or when using RWX block volumes outside of KubeVirt.
|
||||
+func DisableRWXBlockValidation() func(*Driver) error {
|
||||
+ return func(d *Driver) error {
|
||||
+ d.disableRWXBlockValidation = true
|
||||
+ return nil
|
||||
+ }
|
||||
+}
|
||||
+
|
||||
// GetPluginInfo https://github.com/container-storage-interface/spec/blob/v1.9.0/spec.md#getplugininfo
|
||||
func (d Driver) GetPluginInfo(ctx context.Context, req *csi.GetPluginInfoRequest) (*csi.GetPluginInfoResponse, error) {
|
||||
return &csi.GetPluginInfoResponse{
|
||||
@@ -751,6 +764,14 @@ func (d Driver) ControllerPublishVolume(ctx context.Context, req *csi.Controller
|
||||
// ReadWriteMany block volume
|
||||
rwxBlock := req.VolumeCapability.AccessMode.GetMode() == csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER && req.VolumeCapability.GetBlock() != nil
|
||||
|
||||
+ // Validate RWX block attachment to prevent misuse of allow-two-primaries
|
||||
+ if rwxBlock && !d.disableRWXBlockValidation {
|
||||
+ if _, err := utils.ValidateRWXBlockAttachment(ctx, d.kubeClient, d.log, req.GetVolumeId()); err != nil {
|
||||
+ return nil, status.Errorf(codes.FailedPrecondition,
|
||||
+ "ControllerPublishVolume failed for %s: %v", req.GetVolumeId(), err)
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
devPath, err := d.Assignments.Attach(ctx, req.GetVolumeId(), req.GetNodeId(), rwxBlock)
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.Internal,
|
||||
diff --git a/pkg/utils/rwx_validation.go b/pkg/utils/rwx_validation.go
|
||||
new file mode 100644
|
||||
index 00000000..9fe82768
|
||||
--- /dev/null
|
||||
+++ b/pkg/utils/rwx_validation.go
|
||||
@@ -0,0 +1,263 @@
|
||||
+/*
|
||||
+CSI Driver for Linstor
|
||||
+Copyright © 2018 LINBIT USA, LLC
|
||||
+
|
||||
+This program is free software; you can redistribute it and/or modify
|
||||
+it under the terms of the GNU General Public License as published by
|
||||
+the Free Software Foundation; either version 2 of the License, or
|
||||
+(at your option) any later version.
|
||||
+
|
||||
+This program is distributed in the hope that it will be useful,
|
||||
+but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
+GNU General Public License for more details.
|
||||
+
|
||||
+You should have received a copy of the GNU General Public License
|
||||
+along with this program; if not, see <http://www.gnu.org/licenses/>.
|
||||
+*/
|
||||
+
|
||||
+package utils
|
||||
+
|
||||
+import (
|
||||
+ "context"
|
||||
+ "fmt"
|
||||
+
|
||||
+ "github.com/sirupsen/logrus"
|
||||
+ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
+ "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
|
||||
+ "k8s.io/apimachinery/pkg/runtime/schema"
|
||||
+ "k8s.io/client-go/dynamic"
|
||||
+)
|
||||
+
|
||||
+// KubeVirtVMLabel is the label that KubeVirt adds to pods to identify the VM they belong to.
|
||||
+const KubeVirtVMLabel = "vm.kubevirt.io/name"
|
||||
+
|
||||
+// KubeVirtHotplugDiskLabel is the label that KubeVirt adds to hotplug disk pods.
|
||||
+const KubeVirtHotplugDiskLabel = "kubevirt.io"
|
||||
+
|
||||
+// PodGVR is the GroupVersionResource for pods.
|
||||
+var PodGVR = schema.GroupVersionResource{Group: "", Version: "v1", Resource: "pods"}
|
||||
+
|
||||
+// PVGVR is the GroupVersionResource for persistent volumes.
|
||||
+var PVGVR = schema.GroupVersionResource{Group: "", Version: "v1", Resource: "persistentvolumes"}
|
||||
+
|
||||
+// ValidateRWXBlockAttachment checks that RWX block volumes are only used by pods belonging to the same VM.
|
||||
+// This prevents misuse of allow-two-primaries while still permitting live migration.
|
||||
+// Returns the VM name if validation passes, or an error if:
|
||||
+// - Multiple pods from different VMs are trying to use the same volume
|
||||
+// - A pod without the KubeVirt VM label is trying to use a volume already attached elsewhere (strict mode)
|
||||
+// Returns empty string for VM name when no pods are using the volume or validation is skipped.
|
||||
+func ValidateRWXBlockAttachment(ctx context.Context, kubeClient dynamic.Interface, log *logrus.Entry, volumeID string) (string, error) {
|
||||
+ log.WithField("volumeID", volumeID).Info("validateRWXBlockAttachment called")
|
||||
+
|
||||
+ if kubeClient == nil {
|
||||
+ // Not running in Kubernetes, skip validation
|
||||
+ log.Warn("validateRWXBlockAttachment: kubeClient is nil, skipping validation")
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ // Get PV to find PVC reference
|
||||
+ pv, err := kubeClient.Resource(PVGVR).Get(ctx, volumeID, metav1.GetOptions{})
|
||||
+ if err != nil {
|
||||
+ log.WithError(err).Warn("cannot validate RWX attachment: failed to get PV")
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ // Verify that PV's volumeHandle matches the volumeID
|
||||
+ volumeHandle, found, err := unstructured.NestedString(pv.Object, "spec", "csi", "volumeHandle")
|
||||
+ if err != nil {
|
||||
+ log.WithError(err).Warnf("cannot validate RWX attachment: failed to read volumeHandle for PV %s", volumeID)
|
||||
+
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ if !found {
|
||||
+ log.Warnf("cannot validate RWX attachment: volumeHandle not found for PV %s", volumeID)
|
||||
+
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ if volumeHandle != volumeID {
|
||||
+ log.WithFields(logrus.Fields{
|
||||
+ "volumeID": volumeID,
|
||||
+ "volumeHandle": volumeHandle,
|
||||
+ }).Warn("cannot validate RWX attachment: PV volumeHandle does not match volumeID")
|
||||
+
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ // Extract claimRef from PV
|
||||
+ claimRef, found, _ := unstructured.NestedMap(pv.Object, "spec", "claimRef")
|
||||
+ if !found {
|
||||
+ log.Warn("cannot validate RWX attachment: PV has no claimRef")
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ pvcName, _, _ := unstructured.NestedString(claimRef, "name")
|
||||
+ pvcNamespace, _, _ := unstructured.NestedString(claimRef, "namespace")
|
||||
+
|
||||
+ if pvcNamespace == "" || pvcName == "" {
|
||||
+ log.Warn("cannot validate RWX attachment: PVC name or namespace is empty in claimRef")
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ // List all pods in the namespace
|
||||
+ podList, err := kubeClient.Resource(PodGVR).Namespace(pvcNamespace).List(ctx, metav1.ListOptions{})
|
||||
+ if err != nil {
|
||||
+ return "", fmt.Errorf("failed to list pods in namespace %s: %w", pvcNamespace, err)
|
||||
+ }
|
||||
+
|
||||
+ // Filter pods that use this PVC and are in a running/pending state
|
||||
+ type podInfo struct {
|
||||
+ name string
|
||||
+ vmName string
|
||||
+ }
|
||||
+
|
||||
+ var podsUsingPVC []podInfo
|
||||
+
|
||||
+ for _, item := range podList.Items {
|
||||
+ // Get pod phase from status
|
||||
+ phase, _, _ := unstructured.NestedString(item.Object, "status", "phase")
|
||||
+ if phase == "Succeeded" || phase == "Failed" {
|
||||
+ continue
|
||||
+ }
|
||||
+
|
||||
+ // Check if pod uses the PVC
|
||||
+ volumes, found, _ := unstructured.NestedSlice(item.Object, "spec", "volumes")
|
||||
+ if !found {
|
||||
+ continue
|
||||
+ }
|
||||
+
|
||||
+ for _, vol := range volumes {
|
||||
+ volMap, ok := vol.(map[string]interface{})
|
||||
+ if !ok {
|
||||
+ continue
|
||||
+ }
|
||||
+
|
||||
+ claimName, found, _ := unstructured.NestedString(volMap, "persistentVolumeClaim", "claimName")
|
||||
+ if !found || claimName != pvcName {
|
||||
+ continue
|
||||
+ }
|
||||
+
|
||||
+ // Extract VM name, handling both regular and hotplug disk pods
|
||||
+ vmName, err := GetVMNameFromPod(ctx, kubeClient, log, &item)
|
||||
+ if err != nil {
|
||||
+ log.WithError(err).WithField("pod", item.GetName()).Warn("failed to get VM name from pod")
|
||||
+ // Continue with empty vmName - will be caught by strict mode check
|
||||
+ vmName = ""
|
||||
+ }
|
||||
+
|
||||
+ podsUsingPVC = append(podsUsingPVC, podInfo{
|
||||
+ name: item.GetName(),
|
||||
+ vmName: vmName,
|
||||
+ })
|
||||
+
|
||||
+ break
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
+ // If 0 or 1 pod uses the PVC, no conflict possible
|
||||
+ if len(podsUsingPVC) <= 1 {
|
||||
+ // Return VM name if there's exactly one pod
|
||||
+ if len(podsUsingPVC) == 1 {
|
||||
+ log.WithFields(logrus.Fields{
|
||||
+ "volumeID": volumeID,
|
||||
+ "vmName": podsUsingPVC[0].vmName,
|
||||
+ "podCount": 1,
|
||||
+ "pvcNamespace": pvcNamespace,
|
||||
+ "pvcName": pvcName,
|
||||
+ }).Info("validateRWXBlockAttachment: single pod found, returning VM name")
|
||||
+
|
||||
+ return podsUsingPVC[0].vmName, nil
|
||||
+ }
|
||||
+
|
||||
+ log.WithFields(logrus.Fields{
|
||||
+ "volumeID": volumeID,
|
||||
+ "pvcNamespace": pvcNamespace,
|
||||
+ "pvcName": pvcName,
|
||||
+ }).Info("validateRWXBlockAttachment: no pods found using PVC")
|
||||
+
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ // Check that all pods belong to the same VM
|
||||
+ var vmName string
|
||||
+ for _, pod := range podsUsingPVC {
|
||||
+ if pod.vmName == "" {
|
||||
+ // Strict mode: if any pod doesn't have the KubeVirt label and there are multiple pods,
|
||||
+ // deny the attachment
|
||||
+ return "", fmt.Errorf("RWX block volume %s/%s is used by multiple pods but pod %s does not have the %s label; "+
|
||||
+ "RWX block volumes with allow-two-primaries are only supported for KubeVirt live migration",
|
||||
+ pvcNamespace, pvcName, pod.name, KubeVirtVMLabel)
|
||||
+ }
|
||||
+
|
||||
+ if vmName == "" {
|
||||
+ vmName = pod.vmName
|
||||
+ } else if vmName != pod.vmName {
|
||||
+ // Different VMs are trying to use the same volume
|
||||
+ return "", fmt.Errorf("RWX block volume %s/%s is being used by pods from different VMs (%s and %s); "+
|
||||
+ "this is not supported - RWX block volumes with allow-two-primaries are only for live migration of a single VM",
|
||||
+ pvcNamespace, pvcName, vmName, pod.vmName)
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
+ log.WithFields(logrus.Fields{
|
||||
+ "pvcNamespace": pvcNamespace,
|
||||
+ "pvcName": pvcName,
|
||||
+ "vmName": vmName,
|
||||
+ "podCount": len(podsUsingPVC),
|
||||
+ }).Debug("RWX block attachment validated: all pods belong to the same VM (likely live migration)")
|
||||
+
|
||||
+ return vmName, nil
|
||||
+}
|
||||
+
|
||||
+// GetVMNameFromPod extracts the VM name from a pod, handling both regular virt-launcher pods
|
||||
+// and hotplug disk pods (which reference the virt-launcher pod via ownerReferences).
|
||||
+func GetVMNameFromPod(ctx context.Context, kubeClient dynamic.Interface, log *logrus.Entry, pod *unstructured.Unstructured) (string, error) {
|
||||
+ labels := pod.GetLabels()
|
||||
+ if labels == nil {
|
||||
+ return "", nil
|
||||
+ }
|
||||
+
|
||||
+ // Direct case: pod has vm.kubevirt.io/name label (virt-launcher pod)
|
||||
+ if vmName, ok := labels[KubeVirtVMLabel]; ok && vmName != "" {
|
||||
+ return vmName, nil
|
||||
+ }
|
||||
+
|
||||
+ // Hotplug disk case: pod has kubevirt.io: hotplug-disk label
|
||||
+ // Follow ownerReferences to find the virt-launcher pod
|
||||
+ if hotplugValue, ok := labels[KubeVirtHotplugDiskLabel]; ok && hotplugValue == "hotplug-disk" {
|
||||
+ ownerRefs := pod.GetOwnerReferences()
|
||||
+ for _, owner := range ownerRefs {
|
||||
+ if owner.Kind != "Pod" || owner.Controller == nil || !*owner.Controller {
|
||||
+ continue
|
||||
+ }
|
||||
+
|
||||
+ // Get the owner pod (virt-launcher)
|
||||
+ ownerPod, err := kubeClient.Resource(PodGVR).Namespace(pod.GetNamespace()).Get(ctx, owner.Name, metav1.GetOptions{})
|
||||
+ if err != nil {
|
||||
+ return "", fmt.Errorf("failed to get owner pod %s: %w", owner.Name, err)
|
||||
+ }
|
||||
+
|
||||
+ // Extract VM name from owner pod
|
||||
+ ownerLabels := ownerPod.GetLabels()
|
||||
+ if ownerLabels != nil {
|
||||
+ if vmName, ok := ownerLabels[KubeVirtVMLabel]; ok && vmName != "" {
|
||||
+ log.WithFields(logrus.Fields{
|
||||
+ "hotplugPod": pod.GetName(),
|
||||
+ "virtLauncher": owner.Name,
|
||||
+ "vmName": vmName,
|
||||
+ }).Debug("resolved VM name from hotplug disk pod via owner reference")
|
||||
+
|
||||
+ return vmName, nil
|
||||
+ }
|
||||
+ }
|
||||
+
|
||||
+ return "", fmt.Errorf("owner pod %s does not have %s label", owner.Name, KubeVirtVMLabel)
|
||||
+ }
|
||||
+
|
||||
+ return "", fmt.Errorf("hotplug disk pod %s has no controller owner reference", pod.GetName())
|
||||
+ }
|
||||
+
|
||||
+ return "", nil
|
||||
+}
|
||||
diff --git a/pkg/utils/rwx_validation_test.go b/pkg/utils/rwx_validation_test.go
|
||||
new file mode 100644
|
||||
index 00000000..d75690f9
|
||||
--- /dev/null
|
||||
+++ b/pkg/utils/rwx_validation_test.go
|
||||
@@ -0,0 +1,342 @@
|
||||
+/*
|
||||
+CSI Driver for Linstor
|
||||
+Copyright © 2018 LINBIT USA, LLC
|
||||
+
|
||||
+This program is free software; you can redistribute it and/or modify
|
||||
+it under the terms of the GNU General Public License as published by
|
||||
+the Free Software Foundation; either version 2 of the License, or
|
||||
+(at your option) any later version.
|
||||
+
|
||||
+This program is distributed in the hope that it will be useful,
|
||||
+but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
+GNU General Public License for more details.
|
||||
+
|
||||
+You should have received a copy of the GNU General Public License
|
||||
+along with this program; if not, see <http://www.gnu.org/licenses/>.
|
||||
+*/
|
||||
+
|
||||
+package utils
|
||||
+
|
||||
+import (
|
||||
+ "context"
|
||||
+ "testing"
|
||||
+
|
||||
+ "github.com/sirupsen/logrus"
|
||||
+ "github.com/stretchr/testify/assert"
|
||||
+ "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
|
||||
+ "k8s.io/apimachinery/pkg/runtime"
|
||||
+ "k8s.io/apimachinery/pkg/runtime/schema"
|
||||
+ dynamicfake "k8s.io/client-go/dynamic/fake"
|
||||
+)
|
||||
+
|
||||
+func TestValidateRWXBlockAttachment(t *testing.T) {
|
||||
+ testCases := []struct {
|
||||
+ name string
|
||||
+ pods []*unstructured.Unstructured
|
||||
+ pvcName string
|
||||
+ namespace string
|
||||
+ expectError bool
|
||||
+ errorMsg string
|
||||
+ }{
|
||||
+ {
|
||||
+ name: "no pods using PVC",
|
||||
+ pods: []*unstructured.Unstructured{},
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "single pod using PVC",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("pod1", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "two pods same VM (live migration)",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("virt-launcher-vm1-abc", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("virt-launcher-vm1-xyz", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "two pods different VMs (should fail)",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("virt-launcher-vm1-abc", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("virt-launcher-vm2-xyz", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm2"}, "Running"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: true,
|
||||
+ errorMsg: "different VMs",
|
||||
+ },
|
||||
+ {
|
||||
+ name: "pod without KubeVirt label when multiple pods exist (strict mode)",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("pod1", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("pod2", "default", "test-pvc", map[string]string{}, "Running"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: true,
|
||||
+ errorMsg: "does not have the vm.kubevirt.io/name label",
|
||||
+ },
|
||||
+ {
|
||||
+ name: "completed pods should be ignored",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("pod1", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("pod2", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm2"}, "Succeeded"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "failed pods should be ignored",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("pod1", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("pod2", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm2"}, "Failed"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "pods in different namespace should not conflict",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("pod1", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("pod2", "other", "test-pvc", map[string]string{KubeVirtVMLabel: "vm2"}, "Running"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "pods using different PVCs should not conflict",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("pod1", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("pod2", "default", "other-pvc", map[string]string{KubeVirtVMLabel: "vm2"}, "Running"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "three pods from same VM (multi-node live migration scenario)",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("virt-launcher-vm1-a", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("virt-launcher-vm1-b", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createUnstructuredPod("virt-launcher-vm1-c", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Pending"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "hotplug disk pod with virt-launcher (should succeed)",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("virt-launcher-vm1-abc", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createHotplugDiskPod("hp-volume-xyz", "default", "test-pvc", "virt-launcher-vm1-abc", "Running"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: false,
|
||||
+ },
|
||||
+ {
|
||||
+ name: "hotplug disks from different VMs (should fail)",
|
||||
+ pods: []*unstructured.Unstructured{
|
||||
+ createUnstructuredPod("virt-launcher-vm1-abc", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm1"}, "Running"),
|
||||
+ createHotplugDiskPod("hp-volume-vm1", "default", "test-pvc", "virt-launcher-vm1-abc", "Running"),
|
||||
+ createUnstructuredPod("virt-launcher-vm2-xyz", "default", "test-pvc", map[string]string{KubeVirtVMLabel: "vm2"}, "Running"),
|
||||
+ createHotplugDiskPod("hp-volume-vm2", "default", "test-pvc", "virt-launcher-vm2-xyz", "Running"),
|
||||
+ },
|
||||
+ pvcName: "test-pvc",
|
||||
+ namespace: "default",
|
||||
+ expectError: true,
|
||||
+ errorMsg: "different VMs",
|
||||
+ },
|
||||
+ }
|
||||
+
|
||||
+ for _, tc := range testCases {
|
||||
+ t.Run(tc.name, func(t *testing.T) {
|
||||
+ // Create fake dynamic client with test pods and PV
|
||||
+ scheme := runtime.NewScheme()
|
||||
+
|
||||
+ // Create PV object that references the PVC
|
||||
+ pv := createUnstructuredPV("test-volume-id", tc.namespace, tc.pvcName)
|
||||
+
|
||||
+ objects := make([]runtime.Object, 0, len(tc.pods)+1)
|
||||
+ objects = append(objects, pv)
|
||||
+
|
||||
+ for _, pod := range tc.pods {
|
||||
+ objects = append(objects, pod)
|
||||
+ }
|
||||
+
|
||||
+ gvrToListKind := map[schema.GroupVersionResource]string{
|
||||
+ PodGVR: "PodList",
|
||||
+ PVGVR: "PersistentVolumeList",
|
||||
+ }
|
||||
+ client := dynamicfake.NewSimpleDynamicClientWithCustomListKinds(scheme, gvrToListKind, objects...)
|
||||
+
|
||||
+ // Create logger
|
||||
+ logger := logrus.NewEntry(logrus.New())
|
||||
+ logger.Logger.SetLevel(logrus.DebugLevel)
|
||||
+
|
||||
+ // Run validation
|
||||
+ vmName, err := ValidateRWXBlockAttachment(context.Background(), client, logger, "test-volume-id")
|
||||
+
|
||||
+ if tc.expectError {
|
||||
+ assert.Error(t, err)
|
||||
+
|
||||
+ if tc.errorMsg != "" {
|
||||
+ assert.Contains(t, err.Error(), tc.errorMsg)
|
||||
+ }
|
||||
+ } else {
|
||||
+ assert.NoError(t, err)
|
||||
+ // VM name is returned when there are pods using the volume
|
||||
+ if len(tc.pods) > 0 {
|
||||
+ assert.NotEmpty(t, vmName)
|
||||
+ }
|
||||
+ }
|
||||
+ })
|
||||
+ }
|
||||
+}
|
||||
+
|
||||
+func TestValidateRWXBlockAttachmentNoKubeClient(t *testing.T) {
|
||||
+ // When not running in Kubernetes (no client), validation should be skipped
|
||||
+ logger := logrus.NewEntry(logrus.New())
|
||||
+
|
||||
+ vmName, err := ValidateRWXBlockAttachment(context.Background(), nil, logger, "test-volume-id")
|
||||
+ assert.NoError(t, err)
|
||||
+ assert.Empty(t, vmName)
|
||||
+}
|
||||
+
|
||||
+func TestValidateRWXBlockAttachmentPVNotFound(t *testing.T) {
|
||||
+ // When PV is not found, validation should be skipped with warning
|
||||
+ scheme := runtime.NewScheme()
|
||||
+
|
||||
+ gvrToListKind := map[schema.GroupVersionResource]string{
|
||||
+ PodGVR: "PodList",
|
||||
+ PVGVR: "PersistentVolumeList",
|
||||
+ }
|
||||
+ client := dynamicfake.NewSimpleDynamicClientWithCustomListKinds(scheme, gvrToListKind)
|
||||
+
|
||||
+ logger := logrus.NewEntry(logrus.New())
|
||||
+ logger.Logger.SetLevel(logrus.DebugLevel)
|
||||
+
|
||||
+ vmName, err := ValidateRWXBlockAttachment(context.Background(), client, logger, "non-existent-pv")
|
||||
+ assert.NoError(t, err)
|
||||
+ assert.Empty(t, vmName)
|
||||
+}
|
||||
+
|
||||
+// createUnstructuredPod creates an unstructured pod object for testing.
|
||||
+func createUnstructuredPod(name, namespace, pvcName string, labels map[string]string, phase string) *unstructured.Unstructured {
|
||||
+ pod := &unstructured.Unstructured{
|
||||
+ Object: map[string]interface{}{
|
||||
+ "apiVersion": "v1",
|
||||
+ "kind": "Pod",
|
||||
+ "metadata": map[string]interface{}{
|
||||
+ "name": name,
|
||||
+ "namespace": namespace,
|
||||
+ "labels": toStringInterfaceMap(labels),
|
||||
+ },
|
||||
+ "spec": map[string]interface{}{
|
||||
+ "volumes": []interface{}{
|
||||
+ map[string]interface{}{
|
||||
+ "name": "data",
|
||||
+ "persistentVolumeClaim": map[string]interface{}{
|
||||
+ "claimName": pvcName,
|
||||
+ },
|
||||
+ },
|
||||
+ },
|
||||
+ },
|
||||
+ "status": map[string]interface{}{
|
||||
+ "phase": phase,
|
||||
+ },
|
||||
+ },
|
||||
+ }
|
||||
+
|
||||
+ return pod
|
||||
+}
|
||||
+
|
||||
+// createUnstructuredPV creates an unstructured PersistentVolume object for testing.
|
||||
+func createUnstructuredPV(name, pvcNamespace, pvcName string) *unstructured.Unstructured {
|
||||
+ pv := &unstructured.Unstructured{
|
||||
+ Object: map[string]interface{}{
|
||||
+ "apiVersion": "v1",
|
||||
+ "kind": "PersistentVolume",
|
||||
+ "metadata": map[string]interface{}{
|
||||
+ "name": name,
|
||||
+ },
|
||||
+ "spec": map[string]interface{}{
|
||||
+ "claimRef": map[string]interface{}{
|
||||
+ "name": pvcName,
|
||||
+ "namespace": pvcNamespace,
|
||||
+ },
|
||||
+ "csi": map[string]interface{}{
|
||||
+ "volumeHandle": name,
|
||||
+ },
|
||||
+ },
|
||||
+ },
|
||||
+ }
|
||||
+
|
||||
+ return pv
|
||||
+}
|
||||
+
|
||||
+// toStringInterfaceMap converts map[string]string to map[string]interface{}.
|
||||
+func toStringInterfaceMap(m map[string]string) map[string]interface{} {
|
||||
+ result := make(map[string]interface{})
|
||||
+
|
||||
+ for k, v := range m {
|
||||
+ result[k] = v
|
||||
+ }
|
||||
+
|
||||
+ return result
|
||||
+}
|
||||
+
|
||||
+// createHotplugDiskPod creates a hotplug disk pod that references a virt-launcher pod via ownerReferences.
|
||||
+func createHotplugDiskPod(name, namespace, pvcName, ownerPodName, phase string) *unstructured.Unstructured {
|
||||
+ pod := &unstructured.Unstructured{
|
||||
+ Object: map[string]interface{}{
|
||||
+ "apiVersion": "v1",
|
||||
+ "kind": "Pod",
|
||||
+ "metadata": map[string]interface{}{
|
||||
+ "name": name,
|
||||
+ "namespace": namespace,
|
||||
+ "labels": map[string]interface{}{
|
||||
+ "kubevirt.io": "hotplug-disk",
|
||||
+ },
|
||||
+ "ownerReferences": []interface{}{
|
||||
+ map[string]interface{}{
|
||||
+ "apiVersion": "v1",
|
||||
+ "kind": "Pod",
|
||||
+ "name": ownerPodName,
|
||||
+ "controller": true,
|
||||
+ "blockOwnerDeletion": true,
|
||||
+ },
|
||||
+ },
|
||||
+ },
|
||||
+ "spec": map[string]interface{}{
|
||||
+ "volumes": []interface{}{
|
||||
+ map[string]interface{}{
|
||||
+ "name": "data",
|
||||
+ "persistentVolumeClaim": map[string]interface{}{
|
||||
+ "claimName": pvcName,
|
||||
+ },
|
||||
+ },
|
||||
+ },
|
||||
+ },
|
||||
+ "status": map[string]interface{}{
|
||||
+ "phase": phase,
|
||||
+ },
|
||||
+ },
|
||||
+ }
|
||||
+
|
||||
+ return pod
|
||||
+}
|
||||
@@ -21,6 +21,10 @@ spec:
|
||||
|
||||
scrapeInterval: 30s
|
||||
selectAllByDefault: false
|
||||
{{- with .Values.vmagent.inlineScrapeConfig }}
|
||||
inlineScrapeConfig: |
|
||||
{{- . | nindent 4 }}
|
||||
{{- end }}
|
||||
podScrapeNamespaceSelector:
|
||||
matchLabels:
|
||||
namespace.cozystack.io/monitoring: {{ .Release.Namespace }}
|
||||
|
||||
@@ -79,4 +79,8 @@ vmagent:
|
||||
remoteWrite:
|
||||
urls:
|
||||
- http://vminsert-shortterm:8480/insert/0/prometheus
|
||||
- http://vminsert-longterm:8480/insert/0/prometheus
|
||||
- http://vminsert-longterm:8480/insert/0/prometheus
|
||||
## inlineScrapeConfig: |
|
||||
## - job_name: "custom"
|
||||
## static_configs:
|
||||
## - targets: ["my-service:9090"]
|
||||
|
||||
@@ -224,8 +224,8 @@ func buildPostProcessV3(kindSchemas map[string]string) func(*spec3.OpenAPI) (*sp
|
||||
base, ok1 := doc.Components.Schemas[baseRef]
|
||||
list, ok2 := doc.Components.Schemas[baseListRef]
|
||||
stat, ok3 := doc.Components.Schemas[baseStatusRef]
|
||||
if !(ok1 && ok2 && ok3) && len(kindSchemas) > 0 {
|
||||
return doc, fmt.Errorf("base Application* schemas not found")
|
||||
if !(ok1 && ok2 && ok3) {
|
||||
return doc, nil // not the apps GV — nothing to patch
|
||||
}
|
||||
|
||||
// Clone base schemas for each kind
|
||||
@@ -339,8 +339,8 @@ func buildPostProcessV2(kindSchemas map[string]string) func(*spec.Swagger) (*spe
|
||||
base, ok1 := defs[baseRef]
|
||||
list, ok2 := defs[baseListRef]
|
||||
stat, ok3 := defs[baseStatusRef]
|
||||
if !(ok1 && ok2 && ok3) && len(kindSchemas) > 0 {
|
||||
return sw, fmt.Errorf("base Application* schemas not found")
|
||||
if !(ok1 && ok2 && ok3) {
|
||||
return sw, nil // not the apps GV — nothing to patch
|
||||
}
|
||||
|
||||
for kind, raw := range kindSchemas {
|
||||
|
||||
Reference in New Issue
Block a user