Check if suite property is present for debootstarp action and contains
a string. If suite is not present, exit with an error.
Fixes: #238
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Showing the update progress can cause non-interactive shells which do not
support moving the cursor to become confused and not show the correct
output. For instance, GitLab CI ignores any line which attempts to move the
cursor, meaning some output for the apt action is just discarded.
Disable the apt progress from being shown and instead only showing the apt
output log.
Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>
pigz does gzip compression on multiple cores and multiple processors
and there is improvement in compression time on multi core machines.
Use pigz for gzip compression if installed on the host machine.
Fixes: #218
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
If the overlay action fails to copy a file into the target filesystem
(e.g. destination path inside the filesystem doesn't exist), the current
behaviour is to panic, which causes a debos call inside the fakemachine
to panic, which isn't detected by the outer debos call.
This causes the execution of the inner debos call to stop (i.e. the
remaining recipe actions are no longer ran which is expected), but the
postexec commands still run and the overall exection is marked as
successful!
Rework the panics to instead bubble up errors so that any errors when
overlaying files causes the recipe to error out correctly rather than
this unexpected behaviour.
Before this commit is applied, the panic causes the debos call inside the
fakemachine to fail without being trapped by the outer debos call and the
outer debos call to still run the postprocess commands and exit with no
error as if the overlay action was successful:
$ debos tests/overlay-non-existent-destination/overlay-non-existent-destination.yaml
2023/07/05 11:07:24 ==== Overlay file to a non-existent destination ====
2023/07/05 11:07:24 Overlaying tests/overlay-non-existent-destination/overlay-non-existent-destination.yaml on /scratch/root/this/path/does/not/exist
2023/07/05 11:07:24 Failed to copy file tests/overlay-non-existent-destination/overlay-non-existent-destination.yaml: open /scratch/root/this/path/does/not/3277940894: no such file or directory
2023/07/05 12:07:24 ==== run ====
2023/07/05 12:07:24 echo Test | Test
2023/07/05 12:07:24 ==== Recipe done ====
$ echo $?
0
With this commit applied, the execution of the outer debos call stops when
the overlay action fails and the error is correctly bubbled up to the user:
$ debos tests/overlay-non-existent-destination/overlay-non-existent-destination.yaml
2023/07/05 11:08:15 ==== Overlay file to a non-existent destination ====
2023/07/05 11:08:15 Overlaying tests/overlay-non-existent-destination/overlay-non-existent-destination.yaml on /scratch/root/this/path/does/not/exist
2023/07/05 11:08:15 Action `Overlay file to a non-existent destination` failed at stage Run, error: Failed to copy file tests/overlay-non-existent-destination/overlay-non-existent-destination.yaml: open /scratch/root/this/path/does/not/1742738134: no such file or directory
$ echo $?
1
Fixes: #401
Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>
Attempt to overlay a file into a directory in the filesystem which doesn't
exist. The expected behavior is to cause an error.
This test is currently a manual test case and is not hooked into the CI.
Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>
Setting up each required job by hand is annoying; So similar to what we
did with bors have one job that concludes the overall status. As opposed
to bors this *always* runs rather then only when everything is good, as
that allows github to trigger properly.
Signed-off-by: Sjoerd Simons <sjoerd@collabora.com>
402: image_partition_action.go: allow fraction sizes for binary units r=obbardc a=venkata-pyla
The current filter for checking the size unit is not considered the fractions and it is wrongly filtered.
e.g: 10.5GiB is filtered for or converted value as 10.5GB.
Before fix:
When size is 10.5GiB, the image is generated with 10500000000Bytes (wrong)
After fix:
When size is 10.5GiB, the image is generated with 11274289152Bytes (correct)
Suggested-by: Akihiro Suzuki <akihiro27.suzuki@toshiba.co.jp>
Co-authored-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
The current filter for checking the size unit is not considered the
fractions and it is wrongly filtered.
e.g: 10.5GiB is filtered for or converted value as 10.5GB.
Before fix:
When size is 10.5GiB, the image is generated with 10500000000Bytes (wrong)
After fix:
When size is 10.5GiB, the image is generated with 11274289152Bytes (correct)
Suggested-by: Akihiro Suzuki <akihiro27.suzuki@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
397: build(deps): bump github.com/go-debos/fakemachine from 0.0.3 to 0.0.4 r=obbardc a=dependabot[bot]
Bumps [github.com/go-debos/fakemachine](https://github.com/go-debos/fakemachine) from 0.0.3 to 0.0.4.
<details>
<summary>Commits</summary>
<ul>
<li><a href="f31faeb506"><code>f31faeb</code></a> Merge <a href="https://github-redirect.dependabot.com/go-debos/fakemachine/issues/117">#117</a></li>
<li><a href="9311ecad4d"><code>9311eca</code></a> Add CI test to ensure man page is kept up-to-date</li>
<li><a href="24386589a7"><code>2438658</code></a> docs: add manpage</li>
<li><a href="25bc9478cf"><code>25bc947</code></a> docs: correct spelling of fakemachine</li>
<li><a href="f4abe050cb"><code>f4abe05</code></a> Merge <a href="https://github-redirect.dependabot.com/go-debos/fakemachine/issues/127">#127</a></li>
<li><a href="9d84400ddf"><code>9d84400</code></a> Merge <a href="https://github-redirect.dependabot.com/go-debos/fakemachine/issues/132">#132</a></li>
<li><a href="b4b03b72cf"><code>b4b03b7</code></a> Merge <a href="https://github-redirect.dependabot.com/go-debos/fakemachine/issues/73">#73</a></li>
<li><a href="8f8c5e49d2"><code>8f8c5e4</code></a> Add quiet mode</li>
<li><a href="15a20cde72"><code>15a20cd</code></a> Quieten poweroff message</li>
<li><a href="f9281ccd52"><code>f9281cc</code></a> Print startup message outside runner service</li>
<li>Additional commits viewable in <a href="https://github.com/go-debos/fakemachine/compare/v0.0.3...v0.0.4">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- ``@dependabot` rebase` will rebase this PR
- ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it
- ``@dependabot` merge` will merge this PR after your CI passes on it
- ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it
- ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging
- ``@dependabot` reopen` will reopen this PR if it is closed
- ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
390: Limit old suite workaround r=obbardc a=andhe
The workaround for https://github.com/go-debos/debos/issues/361 that was applied in b3c1f76bcc breaks recipes for bookworm and newer.
This change makes the workaround less damaging (should atleast work with official debian releases for the forseeable future), while still far from perfect (any derivate distribution that is bookworm-based will fail).
Co-authored-by: Andreas Henriksson <andreas@fatal.se>
377: Arch support - aka pacstrap, pacman and tests r=obbardc a=evelikov
This PR adds support for creating pacman based images (be that Arch or alike). Similar to the Debian variant we have two important actions - pacstrap and pacman.
- pacstrap, requires pacman.conf and mirrorlist files
- pacman is a simple list of packages, just like apt
This PR supersedes https://github.com/go-debos/debos/pull/276 and https://github.com/go-debos/debos/pull/277, where the most significant changes are within the pacstrap action. In particular, we no longer write the config/mirrorlist files.
Considering that pacman is changing faster than debos, it makes sense to have the files as drop-ins instead of trying to hard-code them within debos itself.
Note: the final commit adds a simple CI file, which is not wired. Tips on how to achieve that would be appreciated.
Co-authored-by: Emil Velikov <emil.velikov@collabora.com>
Co-authored-by: Arnaud Rebillout <arnaud.rebillout@collabora.com>
The --no-fakemachine tests currently fail when the test recipe
uses the apt actions with:
2022/12/07 15:22:25 apt clean | Failed to mount cgroup (type cgroup) on /tmp/unifiedANLvrW (MS_NOSUID|MS_NODEV|MS_NOEXEC "none,name=systemd,xattr"): Operation not permitted
For whatever reason the docker container hasn't figured out
the correct cgroup heirarchy from the GitHub actiosn
environment. Fix it by forcing unified cgroup heirarchy in
the container used for the tests.
Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>
Nothing special here - a simple test.yaml akin to the existing ones.
The pacman.conf file is effectively vanilla config that comes with the
pacman package, while the mirrorlist file is a list of UK https mirrors
that I've been using for years.
Note that there is no canonical mirror for Arch. The top-level/tier 1
Arch server should _not_ be used as per the official recommendation.
Instead we use the mirrors set in the official arch docker tooling.
v2:
- Also install archlinux-keyring and makepkg
- Use arch + backend=qemu only testing matrix
v3:
- Use the same mirrors as the Arch docker images tooling
- Build and install pacman locally, until we get an official package
v4:
- Remove the local pacman build - it's available in backports
v5:
- Explicitly pull arch-install-scripts from backports, normal one lack
pacstrap
v6:
- Manually pull the latest keyring - Debian one is outdated
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Add a pacman action, whereby pacman is the Arch package manager akin to
apt for Debian. This allows us to install packages, after the initial
pacstrap action (or equivalent).
Akin to the pacstrap action - there is a Debian package for pacman. Plus
upstream pacman continuously builds and tests pacman against Arch,
Fedora and Debian.
Signed-off-by: Arnaud Rebillout <arnaud.rebillout@collabora.com>
v2: [Ed]
- Drop --color never as it is implicit
- Drop --noprogressbar likewise
- Reorder remaining arguments
- Style tweaks
v3: [Emil]
- Run the lot through gofmt -s -w
- Commit message
- Sort the action alphabetically
v4: [Emil]
- Drop quotes around pacman
Signed-off-by: Ed Smith <ed.smith@collabora.com>
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
---
Similar to the pacstrap action, the formatting is a bit weird props to
gofmt. Happy to drop that hunk.
Add a pacstrap action, which is the Arch variant of debootstrap.
In other words, this allows us to bootstrap an Arch image.
Bear in mind that the tool pacstrap is a thin wrapper around the pacman
package manager. Both of those are separate projects and are available
as Debian packages, in case one wants to give them a try.
This is loosely based on the work by Arnaud Rebillout
Note: the action works only within a fakemachine instance, making it
work outside is doable but left for a later day.
v2:
- drop pacman-key capitalisation
v3:
- drop pacstrap capitalisation
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
---
Some of the changes made:
- Commit message
- Remove all pacman.conf and mirrorlist generation
- Use drop-in user files for ^^
- Remove custom flags passed to pacman-key, pacstrap
- Add Verify hook for validation
- Sort the action
- Run through gofmt -s -w
- Add PreNoMachine() stage
NOTE: the funky formatting of the pacstrap_action.do header is due to
gofmt. Can drop those changes if people prefer.
We have issues with the CI workflow randomly failing
when users submit pull requests. It's not great to
block features added by users on these failures; let's
run the CI more frequently such that these issues are
caught earlier.
Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>
282: actions: overlay: Log using action log functions r=obbardc a=obbardc
Currently the overlay action prints directly to stdout, which
ends up not following the standard output format the other
actions follow. Let's clean this up by using the action's
existing logging functions rather than printing directly
to stdout.
Signed-off-by: Christopher Obbard <chris.obbard@collabora.com>
Co-authored-by: Christopher Obbard <chris.obbard@collabora.com>
383: tests: process json files throgh jq prior to comparing r=obbardc a=evelikov
There are various whitespace changes that can occur in json files. Use the same pretty-format in both files to ensure the output is stable and consistent.
Note: we need to explicitly call "bash -c" here otherwise, we'll end up with pure POSIX shell (sh), where anonymous pipes are not a thing.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
Co-authored-by: Emil Velikov <emil.velikov@collabora.com>
There are various whitespace changes that can occur in json files. Use
the same pretty-format in both files to ensure the output is stable and
consistent.
Note: we need to explicitly call "bash -c" here otherwise, we'll end up
with pure POSIX shell (sh), where anonymous pipes are not a thing.
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
381: image-partition: add unlock() in the error paths r=obbardc a=evelikov
Earlier commit added image locking around the partition handling. Yet we forgot to unlock in the error path - fix that.
Fixes: 1002e0a ("image-partition: Minimize the image lock scope")
Co-authored-by: Emil Velikov <emil.velikov@collabora.com>
Earlier commit added image locking around the partition handling. Yet we
forgot to unlock in the error path - fix that.
v2: use defer - this may lead to File.Close() being called twice, but
internally golang is smart enough to only call `man 3 close` as needed
Fixes: 1002e0a ("image-partition: Minimize the image lock scope")
Signed-off-by: Emil Velikov <emil.velikov@collabora.com>
371: docs: RawAction: list possible values for origin r=obbardc a=jakob-tsd
Getting the raw action to work with an actual file was somewhat confusing for me.
Looking at the code, the origin is searched for in context.Origins, which is only set in three places:
$ git grep Origins
download_action.go: context.Origins[d.Name] = originPath
filesystem_deploy_action.go: context.Origins["filesystem"] = context.ImageMntDir
ostree_deploy_action.go: context.Origins["filesystem"] = context.ImageMntDir
In addition, "recipe" is special-cased in DebosContext.Origin.
List all four options to hopefully make it less confusing for the next person.
373: build(deps): bump github.com/stretchr/testify from 1.8.0 to 1.8.1 r=obbardc a=dependabot[bot]
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.0 to 1.8.1.
<details>
<summary>Commits</summary>
<ul>
<li><a href="b747d7c5f8"><code>b747d7c</code></a> Bump github.com/stretchr/objx from 0.4.0 to 0.5.0 (<a href="https://github-redirect.dependabot.com/stretchr/testify/issues/1283">#1283</a>)</li>
<li>See full diff in <a href="https://github.com/stretchr/testify/compare/v1.8.0...v1.8.1">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting ``@dependabot` rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- ``@dependabot` rebase` will rebase this PR
- ``@dependabot` recreate` will recreate this PR, overwriting any edits that have been made to it
- ``@dependabot` merge` will merge this PR after your CI passes on it
- ``@dependabot` squash and merge` will squash and merge this PR after your CI passes on it
- ``@dependabot` cancel merge` will cancel a previously requested merge and block automerging
- ``@dependabot` reopen` will reopen this PR if it is closed
- ``@dependabot` close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- ``@dependabot` ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- ``@dependabot` ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Co-authored-by: Jakob Unterwurzacher <jakob.unterwurzacher@theobroma-systems.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
