789 Commits

Author SHA1 Message Date
Dmitriy Matrenichev
e26d0043e0 chore: code cleanup
More usage of slices package, less usage of package sort.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-11-14 12:25:56 +03:00
Noel Georgi
682718d4c9 fix: use imager incoming version for extension validation
Use the version coming from imager to validate extension constraints.

Part of : #9694

Signed-off-by: Noel Georgi <git@frezbo.dev>
2024-11-12 15:26:59 +05:30
Noel Georgi
2001167058 chore(ci): save support zip always after tests
Save `support.zip` always, also use a different folder for saving logs,
so we can save artifacts of multi cluster tests.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2024-11-08 19:18:06 +05:30
Andrey Smirnov
05c620957c feat: allow extra mounts for docker-based talosctl cluster create
Fixes #9607

Use docker CLI syntax, support any kind of mounts supported by docker
CLI.

Also drop modules from `talos` container image, as it's useless to
provide modules in container mode.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-11-05 16:10:08 +04:00
Andrey Smirnov
d39393879a fix: rework the 'metal-iso' config acquisition
Fixes #9538

Re-do the implementation by using the volume management primitives, so
that we can avoid/skip old code. This should fix all issues related to
the partition/whole disk.

Fix issues in the volume management (exposed, as we haven't used it this
way before).

Build a test case in `talosctl cluster create` to inject machine config
via `metal-iso`.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-10-24 16:18:50 +04:00
Noel Georgi
1b22df48a4 chore: support debug shell for advanced development
Support dropping into a very minimal debug shell.

```bash
sudo -E --preserve-env=HOME _out/talosctl-linux-amd64 cluster create --provisioner=qemu $REGISTRY_MIRROR_FLAGS --controlplanes=1 --workers=0 --with-bootloader=false --with-debug-shell
```

Co-authored-by: Dmitry Sharshakov <dmitry.sharshakov@siderolabs.com>
Signed-off-by: Noel Georgi <git@frezbo.dev>
Signed-off-by: Dmitry Sharshakov <dmitry.sharshakov@siderolabs.com>
2024-10-19 16:56:24 +02:00
Andrey Smirnov
e0434d77d7 feat: update dependencies
Bring in new tools, pkgs, update Go dependencies and others.

In preparation for Talos 1.9.0-alpha.0.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-10-17 22:12:50 +04:00
Andrey Smirnov
bc4c21f41a test: add json logs test environment
Add an option to `talosctl cluster create` to start a JSON log receiver,
and enabled it optionally.

Enable in `integration-qemu`.

See #9510

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-10-17 16:23:26 +04:00
Dmitriy Matrenichev
59a78da42c chore: add proto-codec/codec
Unify usage of proto codec v2 across our projects.
Bump grpc library to 1.67.1 and ensure that we it still works with HTTP/2 ALPN value changes.

For https://github.com/siderolabs/talos/issues/9404

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-10-17 00:12:42 +03:00
Dmitriy Matrenichev
7ff1cedfe3 chore: update siderolabs/crypto module and return proper ALPN
Fixes #9463

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-10-16 22:12:49 +03:00
Noel Georgi
a859cff364 chore: use virtio driver for disks in arm64
ARM64 doesn't support `ide` as a disk driver for disks, use `virtio`
instead.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2024-10-16 17:43:13 +05:30
Dmitriy Matrenichev
4d902021bb fix: do not use pflag csv comma reader for config-patch
Modules pflag and cobra use csv.Reader for `StringSliceVar` method. This doesn't work well with JSON, and we do not need this at all.
Drop it.

Fixes #9493

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-10-14 11:31:23 +03:00
Andrey Smirnov
908fd8789c feat: support cgroup deep analysis in talosctl
The new command `talosctl cgroups` fetches cgroups snapshot from the
machine, parses it fully, enhances with additional information (e.g.
resolves pod names), and presents a customizable view of cgroups
configuration (e.g. limits) and current consumption.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-09-30 18:57:12 +04:00
Andrey Smirnov
392c4798f0 feat: prepare for Talos 1.9
Update tools, pkgs, extras.

Brings in Go 1.23.1, Linux 6.6.52, new xfsprogs, etc.

Fork docs.

Add new version contract, etc.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-09-27 21:23:48 +04:00
Andrey Smirnov
8166a58b36 fix: filter out non-printable characters in process line
Otherwise the output might be distorted by characters like `\n`.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-09-19 21:19:47 +04:00
Andrey Smirnov
8d6884a8e2 test: add a test for inline machine config trusted roots
Run SideroLink API server via TLS with self-signed certificate, inject
that certificate into Talos via `talos.config.inline=`.

Fix a couple of place where our special TLS root CA provider supporting
reloading on the fly was not used.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-09-12 21:44:36 +04:00
Dmitriy Matrenichev
cd7c682662 chore: disallow duplicate documents on decoder level
Required for #9275

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-09-06 15:50:13 +03:00
Andrey Smirnov
3038ccfa88 feat: add configuration for EPHEMERAL volume
Fixes #9261

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-09-06 14:11:35 +04:00
Artem Chernyshev
faffa4c3f1 fix: never unarchive initramfs when loading boot assets in talosctl
The initramfs unarchive won't work as it's extension is `xz` while the
actual compression is `zst`.

Signed-off-by: Artem Chernyshev <artem.chernyshev@talos-systems.com>
2024-09-05 20:09:57 +03:00
Andrey Smirnov
6f7c3a8e5c fix: build of talosctl on non-Linux arches
Move META constants out to machinery, and fix up imports. The internal
`pkg/meta` package shold not be consumed in public-facing commands.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-08-30 22:17:38 +04:00
Andrey Smirnov
b453385bd9 feat: support volume configuration, provisioning, etc
This implements the first round of changes, replacing the volume backend
with the new implementation, while keeping most of the external
interfaces intact.

See #8367

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-08-30 18:32:34 +04:00
Andrey Smirnov
be2ebf6b4d chore: bump dependencies
Update tools, pkgs, extras, Go dependencies, Go tools, etc.

Linux 6.6.47 and containerd 2.0.0-rc.4.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-08-29 20:44:37 +04:00
Noel Georgi
19a44c2b0b chore: drop console ttyS0 argument
Drop `console=ttyS0` argument for metal images/installer.

`console=ttyS0` causes lot of issues with bare metal hardware when
trying to use a physical serial port.

Ref:

* https://bugzilla.redhat.com/show_bug.cgi?id=1839923
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=763601;msg=17
* https://www.kernel.org/doc/html/latest/admin-guide/serial-console.html
* https://github.com/coreos/fedora-coreos-tracker/issues/567

Fixes: #8695
Fixes: #8657
Fixes: #8127

Signed-off-by: Noel Georgi <git@frezbo.dev>
2024-08-27 22:24:59 +05:30
Dmitry Sharshakov
4834a61a8e feat: report SELinux labels
This will be useful for debugging SELinux implementation. Make API report other xattrs for further development like IMA/EVM

Signed-off-by: Dmitry Sharshakov <dmitry.sharshakov@siderolabs.com>
2024-08-26 16:19:38 +03:00
Noel Georgi
36f83eea9f chore: make qemu check flag consistent with code
Restructure code as per changes from #9198.

This makes the flag name to be in sync with what it actually does.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2024-08-20 20:33:56 +05:30
Andrey Smirnov
61a1c946bf feat: bundle (some) CNI plugins with Talos core
Fixes https://github.com/siderolabs/extensions/issues/448

Bundle some CNI standard plugins plus Flannel CNI plugin (as Flannel is
the default CNI in Talos) in the Talos `initramfs`.

With this change, no plugin install is required, so the `install-cni`
step is dropped from the Flannel default manifest.

The bundled plugins:

```
$ talosctl -n 172.20.0.2 ls -lH /opt/cni/bin/
NODE         MODE         UID   GID   SIZE(B)   LASTMOD       NAME
172.20.0.2   drwxr-xr-x   0     0     109 B     7 hours ago   .
172.20.0.2   -rwxr-xr-x   0     0     3.2 MB    7 hours ago   bridge
172.20.0.2   -rwxr-xr-x   0     0     3.3 MB    7 hours ago   firewall
172.20.0.2   -rwxr-xr-x   0     0     2.4 MB    7 hours ago   flannel
172.20.0.2   -rwxr-xr-x   0     0     2.4 MB    7 hours ago   host-local
172.20.0.2   -rwxr-xr-x   0     0     2.4 MB    7 hours ago   loopback
172.20.0.2   -rwxr-xr-x   0     0     2.8 MB    7 hours ago   portmap
```

The `initramfs` for amd64 grows 67 -> 73 MiB with this change.

The path `/opt/cni/bin` is still an overlay mount, so extra plugins can
be dropped to this directory (no change here).

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-08-14 14:33:18 +04:00
Jean-Francois Roy
fd54dc191d feat(talosctl): append microsoft secure boot certs
This patch adds a flag to `secureboot.database.Generate` to append the
Microsoft UEFI secure boot DB and KEK certificates to the appropriate
ESLs, in addition to complimentary command line flags.

This patch also includes a copy of said Microsoft certificates. The
certificates are downloaded from an official Microsoft repo.

Signed-off-by: Jean-Francois Roy <jf@devklog.net>
Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-07-22 14:15:42 +04:00
Andrey Smirnov
1b8c9ccbb0 fix: enforce secureboot enroll option only for supported releases
Follow up for #9005

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-07-17 14:16:26 +04:00
Andrey Smirnov
b07338f547 feat: provide machine config document to update trusted CA roots
Fixes #8867

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-07-12 19:28:31 +04:00
Andrey Smirnov
cf5effabb2 feat: provide an option to enforce SecureBoot for TPM enrollment
Fixes #8995

There is no security impact, as the actual SecureBoot
state/configuration is measured into the PCR 7 and the disk encryption
key unsealing is tied to this value.

This is more to provide a way to avoid accidentally encrypting to the
TPM while SecureBoot is not enabled.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-07-11 22:21:47 +04:00
Noel Georgi
a727a1d97a chore: make using action tracker easier
Refactor so that action tracker accepts an interface.

Signed-off-by: Noel Georgi <git@frezbo.dev>
2024-07-11 21:51:29 +05:30
Dmitriy Matrenichev
dad9c40c73 chore: simplify code
- replace `interface{}` with `any` using `gofmt -r 'interface{} -> any -w'`
- replace `a = []T{}` with `var a []T` where possible.
- replace `a = []T{}` with `a = make([]T, 0, len(b))` where possible.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-07-08 18:14:00 +03:00
Dmitriy Matrenichev
d9db360ab4 fix: properly output multi-doc machine config in get mc
For `get mc -o json|yaml` we pretend that `spec` field is string and not an actual yaml map. That way you
can see the full spec in unformatted view using `talosctl -n <node> get mc -o yaml` or formatted using
`talosctl -n <node> get mc -o yaml | yq .spec`.

`edit mc` command is unaffected.

Fixes #8687

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-07-08 16:22:55 +03:00
Andrey Smirnov
be35f380cc chore: update pkgs/tools/extras
This brings in Go 1.22.5 and new Flannel CNI plugin.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-07-03 20:38:55 +04:00
Dmitriy Matrenichev
2d054ad355 chore: handle documents diff in apply-config dry run
Before this PR diff generator only diffed the v1alpha1 config and nothing else. With this PR it also takes
separate docs into the account.

```shell
~ > <editor> controlplane.yaml
~ > talosctl -n talos-default-controlplane-1  apply-config --file controlplane.yaml --dry-run
Dry run summary:
Applied configuration without a reboot (skipped in dry-run).
Config diff:
No changes.
Documents diff:
[]config.Document{
+	&runtime.KmsgLogV1Alpha1{
+		Meta:       meta.Meta{MetaAPIVersion: "v1alpha1", MetaKind: "KmsgLogConfig"},
+		MetaName:   "omni-kmsg",
+		KmsgLogURL: s"tcp://[fdae:41e4:649b:9303::1]:8092",
+	},
}
~ > talosctl -n talos-default-controlplane-1  apply-config --file controlplane.yaml
Applied configuration without a reboot
~ >
~ >
~ >
~ > <editor> controlplane.yaml
~ > talosctl -n talos-default-controlplane-1  apply-config --file controlplane.yaml --dry-run
Dry run summary:
Applied configuration without a reboot (skipped in dry-run).
Config diff:
No changes.
Documents diff:
[]config.Document{
	&runtime.KmsgLogV1Alpha1{Meta: {MetaAPIVersion: "v1alpha1", MetaKind: "KmsgLogConfig"}, MetaName: "omni-kmsg", KmsgLogURL: {URL: &{Scheme: "tcp", Host: "[fdae:41e4:649b:9303::1]:8092"}}},
+	&network.DefaultActionConfigV1Alpha1{
+		Meta:    meta.Meta{MetaAPIVersion: "v1alpha1", MetaKind: "NetworkDefaultActionConfig"},
+		Ingress: s"block",
+	},
}
```

Closes #8885

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-06-27 21:15:36 +03:00
Andrey Smirnov
7fcb521a6a feat: use hydrophone instead of sonobuoy
Fixes #8790

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-06-11 16:51:45 +04:00
Andrey Smirnov
f07b79f4a8 feat: provide disk detection based on new blockdevices
Uses go-siderolabs/go-blockdevice/v2 for all the hard parts,
provides new resource `Disk` which describes all disks in the system.

Additional resource `SystemDisk` always point to the system disk (based
on the location of `META` partition).

The `Disks` API (and `talosctl disks`) provides a view now into the
`talosctl get disks` to keep backwards compatibility.

QEMU provisioner can now create extra disks of various types: IDE, AHCI,
SCSI, NVME, this allows to test detection properly.

The new resource will be the foundation for volume provisioning (to pick
up the disk to provision the volume on).

Example:

```
talosctl -n 172.20.0.5 get disks
NODE         NAMESPACE   TYPE   ID        VERSION   SIZE          READ ONLY   TRANSPORT   ROTATIONAL   WWID                                                               MODEL            SERIAL
172.20.0.5   runtime     Disk   loop0     1         65568768      true
172.20.0.5   runtime     Disk   nvme0n1   1         10485760000   false       nvme                     nvme.1b36-6465616462656566-51454d55204e564d65204374726c-00000001   QEMU NVMe Ctrl   deadbeef
172.20.0.5   runtime     Disk   sda       1         10485760000   false       virtio      true                                                                            QEMU HARDDISK
172.20.0.5   runtime     Disk   sdb       1         10485760000   false       sata        true         t10.ATA     QEMU HARDDISK                           QM00013        QEMU HARDDISK
172.20.0.5   runtime     Disk   sdc       1         10485760000   false       sata        true         t10.ATA     QEMU HARDDISK                           QM00001        QEMU HARDDISK
172.20.0.5   runtime     Disk   vda       1         12884901888   false       virtio      true
```

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-06-07 20:18:32 +04:00
Andrey Smirnov
41f92e0ba4 chore: update Go to 1.22.4, other updates
Bump go modules, adjust the code.

New linter warnings.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-06-05 20:59:52 +04:00
Andrey Smirnov
82d9cd3229 fix: add upgrade errata for arm64/zboot kernels
Fixes #8854

Talos 1.8.0 instroduces EFI ZBoot compression, and kexec from 1.7.0 to
compressed kernel doesn't work.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-06-03 20:10:24 +04:00
Dmitriy Matrenichev
eb510d9fdf chore: require enabled bootloader for docker provisioner
Otherwise, it doesn't make sense.

Signed-off-by: Dmitriy Matrenichev <dmitry.matrenichev@siderolabs.com>
2024-05-24 23:15:04 +03:00
Andrey Smirnov
92a274e9a0 fix: workaround problems with udevd races
When `udevd` rescans block device partitions while Talos is doing
partitions, it might be that Talos can hit the following error
while trying to open/mount a partition:

```
no such device or address
```

Previous attempts to fix that were using `ENODEV`, while the proper code
is `ENXIO`.

Also take exclusive lock while working with user disks to prevent
concurrent udevd rescan.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-05-24 21:22:07 +04:00
Andrey Smirnov
e1711cd3c9 chore: stop using containerd package for cri namespace
In containerd 2.0 source tree, this constant is under `internal`, so we
can't import it directly.

So instead re-declare it as a Talos constant.

Doing this multi-staged, as `go-talos-support` is using it as well, and
to update it to stop importing old containerd library I need first to
declare the constant in Talos source tree.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-05-21 17:45:52 +04:00
Artem Chernyshev
42ac5cd0c2 fix: check for nil machine config during installation
Otherwise we get `nil reference` exception during maintenance mode
upgrade with partial machine configs.

Signed-off-by: Artem Chernyshev <artem.chernyshev@talos-systems.com>
2024-05-08 16:34:55 +03:00
Utku Ozdemir
478b862b4c fix: do not fail cli action tracker when boot id cannot be read
If the `reboot/reset/shutdown/upgrade` action tracker cannot read the boot ID from the node under `/proc/sys/kernel/random/boot_id` due to insufficient permissions (e.g., when `talosctl reboot` is used over Omni), fall back to skipping boot ID check instead of hard-failing.

Closes siderolabs/talos#7197.

Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
2024-05-07 13:51:28 +02:00
Andrey Smirnov
4c0c626b78 feat: use zstd compression in place of xz
Initramfs and kernel are compressed with zstd.

Extensions are compressed with zstd for Talos 1.8+.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-04-29 18:09:12 +04:00
Utku Ozdemir
dd1d279daa fix: allow more flags in talosctl cluster create --input-dir
Some of the flags passed to `talosctl cluster create` were failing the input validation due to being incorrectly marked as mutually exclusive with the `--input-dir` flag.

Clean up the check to allow passing all flags along with the `--input-dir` flag if those flags impact the provisioning process in any way (i.e., not solely used in generating machine config).

Additionally, replace the mutual exclusion checks with Cobra's built-in function for that.

Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
2024-04-24 10:49:24 +02:00
Utku Ozdemir
0821b9c50b feat: add --non-masquerade-cidrs flag to talosctl cluster create
Allow skipping NAT for the given destinations from a cluster network. This option makes it possible to form an etcd cluster from clusters in different networks created by running `talosctl cluster create` command multiple times using different CIDRs: they simply should have the CIDR of the other clusters passed with `--non-masquerade-cidrs`.

Signed-off-by: Utku Ozdemir <utku.ozdemir@siderolabs.com>
2024-04-23 13:30:13 +02:00
Dmitry Sharshakov
6fbd1263cc feat: report process MAC labels
This will be useful for debugging process access rights once we start implementing SELinux

Signed-off-by: Dmitry Sharshakov <dmitry.sharshakov@siderolabs.com>
2024-04-22 18:16:33 +03:00
Andrey Smirnov
0a785802ea fix: overlay installer operations
1. Use overlay installer to build the `cmdline` when running in
   install/upgrade mode.

2. Pull down the overlay installer with the arch specific to the
   installer being generated, vs. the arch of the `imager`.

3. Print a message when running an overlay installer.

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
2024-04-16 20:07:44 +04:00
Artem Chernyshev
3dd1f4e88c chore: extract pkg/imager/quirks to pkg/machinery
To make it possible to use it without pulling the whole Talos.

Signed-off-by: Artem Chernyshev <artem.chernyshev@talos-systems.com>
2024-04-15 21:37:47 +03:00