* [QT-436] Pseudo random artifact test scenarios (#18056)
Introducing a new approach to testing Vault artifacts before merge
and after merge/notorization/signing. Rather than run a few static
scenarios across the artifacts, we now have the ability to run a
pseudo random sample of scenarios across many different build artifacts.
We've added 20 possible scenarios for the AMD64 and ARM64 binary
bundles, which we've broken into five test groups. On any given push to
a pull request branch, we will now choose a random test group and
execute its corresponding scenarios against the resulting build
artifacts. This gives us greater test coverage but lets us split the
verification across many different pull requests.
The post-merge release testing pipeline behaves in a similar fashion,
however, the artifacts that we use for testing have been notarized and
signed prior to testing. We've also reduce the number of groups so that
we run more scenarios after merge to a release branch.
We intend to take what we've learned building this in Github Actions and
roll it into an easier to use feature that is native to Enos. Until then,
we'll have to manually add scenarios to each matrix file and manually
number the test group. It's important to note that Github requires every
matrix to include at least one vector, so every artifact that is being
tested must include a single scenario in order for all workflows to pass
and thus satisfy branch merge requirements.
* Add support for different artifact types to enos-run
* Add support for different runner type to enos-run
* Add arm64 scenarios to build matrix
* Expand build matrices to include different variants
* Update Consul versions in Enos scenarios and matrices
* Refactor enos-run environment
* Add minimum version filtering support to enos-run. This allows us to
automatically exclude scenarios that require a more recent version of
Vault
* Add maximum version filtering support to enos-run. This allows us to
automatically exclude scenarios that require an older version of
Vault
* Fix Node 12 deprecation warnings
* Rename enos-verify-stable to enos-release-testing-oss
* Convert artifactory matrix into enos-release-testing-oss matrices
* Add all Vault editions to Enos scenario matrices
* Fix verify version with complex Vault edition metadata
* Rename the crt-builder to ci-helper
* Add more version helpers to ci-helper and Makefile
* Update CODEOWNERS for quality team
* Add support for filtering matrices by group and version constraints
* Add support for pseudo random test scenario execution
Backport
* Fix most `gofumpt` linter errors
Signed-off-by: Ryan Cragun <me@ryan.ec>
* [QT-358] Unify CRT and local builder workflows (#17766)
Here we make the following major changes:
* Centralize CRT builder logic into a script utility so that we can share the
logic for building artifacts in CI or locally.
* Simplify the build workflow by calling a reusable workflow many times
instead of repeating the contents.
* Create a workflow that validates whether or not the build workflow and all
child workflows have succeeded to allow for merge protection.
Motivation
* We need branch requirements for the build workflow and all subsequent
integration tests (QT-353)
* We need to ensure that the Enos local builder works (QT-558)
* Debugging build failures can be difficult because one has to hand craft the
steps to recreate the build
* Merge conflicts between Vault OSS and Vault ENT build workflows are quite
painful. As the build workflow must be the same file and name we'll reduce
what is contained in each that is unique. Implementations of building
will be unique per edition so we don't have to worry about conflict
resolution.
* Since we're going to be touching the build workflow to do the first two
items we might as well try and improve those other issues at the same time
to reduce the overhead of backports and conflicts.
Considerations
* Build logic for Vault OSS and Vault ENT differs
* The Enos local builder was duplicating a lot of what we did in the CRT build
workflow
* Version and other artifact metadata has been an issue before. Debugging it
has been tedious and error prone.
* The build workflow is full of brittle copy and paste that is hard to
understand, especially for all of the release editions in Vault Enterprise
* Branch check requirements for workflows are incredibly painful to use for
workflows that are dynamic or change often. The required workflows have to be
configured in Github settings by administrators. They would also prevent us
from having simple docs PRs since required integration workflows always have
to run to satisfy branch requirements.
* Doormat credentials requirements that are coming will require us to modify
which event types trigger workflows. This changes those ahead of time since
we're doing so much to build workflow. The only noticeable impact will be
that the build workflow no longer runs on pushes to non-main or release
branches. In order to test other branches it requires a workflow_dispatch
from the Actions tab or a pull request.
Solutions
* Centralize the logic that determines build metadata and creates releasable
Vault artifacts. Instead of cargo-culting logic multiple times in the build
workflow and the Enos local modules, we now have a crt-builder script which
determines build metadata and also handles building the UI, Vault, and the
package bundle. There are make targets for all of the available sub-commands.
Now what we use in the pipeline is the same thing as the local builder, and
it can be executed locally by developers. The crt-builder script works in OSS
and Enterprise so we will never have to deal with them being divergent or with
special casing things in the build workflow.
* Refactor the bulk of the Vault building into a reusable workflow that we can
call multiple times. This allows us to define Vault builds in a much simpler
manner and makes resolving merge conflicts much easier.
* Rather than trying to maintain a list and manually configure the branch check
requirements for build, we'll trigger a single workflow that uses the github
event system to determine if the build workflow (all of the sub-workflows
included) have passed. We'll then create branch restrictions on that single
workflow down the line.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Disable the fail-fast feature for the matrix strategy. This prevents a
single failure to cascade to all jobs in the matrix. This prevents cases
where other jobs fail before they're able to clean up resources.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Add our initial Enos integration tests to Vault. The Enos scenario
workflow will automatically be run on branches that are created from the
`hashicorp/vault` repository. See the README.md in ./enos a full description
of how to compose and execute scenarios locally.
* Simplify the metadata build workflow jobs
* Automatically determine the Go version from go.mod
* Add formatting check for Enos integration scenarios
* Add Enos smoke and upgrade integration scenarios
* Add Consul backend matrix support
* Add Ubuntu and RHEL distro support
* Add Vault edition support
* Add Vault architecture support
* Add Vault builder support
* Add Vault Shamir and awskms auto-unseal support
* Add Raft storage support
* Add Raft auto-join voter verification
* Add Vault version verification
* Add Vault seal verification
* Add in-place upgrade support for all variants
* Add four scenario variants to CI. These test a maximal distribution of
the aforementioned variants with the `linux/amd64` Vault install
bundle.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Rebecca Willett <rwillett@hashicorp.com>
Co-authored-by: Jaymala <jaymalasinha@gmail.com>
If we don't guard against pull_request being null, we do a lot of extra
checkout and path filtering, and it ends up putting everything in the UI
board.
I tested this in another repo, and it seems to behave correctly.
Add Open Source project workflow
This will help us triage open source issues into our various internal
project boards.
I tested this on a separate repo, and it seems to work.
Update Go to 1.18
From 1.17.12
1.18.5 was just released, but not all packages have been updated, so I
went with 1.18.4
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
This is from https://github.com/hashicorp/backport-assistant/pull/40, so that we backport only the merge commit instead of the individual PR commits. This requires that the PR have been merged using the squash commit strategy, which is our policy. (#15571)
* Copy UBI Dockerfile into Vault
This Dockerfile was modeled off of the existing Alpine Dockerfile (in
this repo) and the external Dockerfile from the docker-vault repo:
> https://github.com/hashicorp/docker-vault/blob/master/ubi/Dockerfile
We also import the UBI-specific docker-entrypoint.sh, as certain
RHEL/Alpine changes (like interpreter) require a separate entry script.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add UBI build to CRT pipeline
Also adds workflow_dispatch to the CRT pipeline, to allow manually
triggering CRT from PRs, when desired.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update Dockerfile
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
* Update Dockerfile
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
* Update Dockerfile
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
* Update Dockerfile
* Update Dockerfile
* Update build.yml
Allow for both push to arbitrary branch plus workflow dispatch, per Newsletter article.
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
* add BuildDate to version base
* populate BuildDate with ldflags
* include BuildDate in FullVersionNumber
* add BuildDate to seal-status and associated status cmd
* extend core/versions entries to include BuildDate
* include BuildDate in version-history API and CLI
* fix version history tests
* fix sys status tests
* fix TestStatusFormat
* remove extraneous LD_FLAGS from build.sh
* add BuildDate to build.bat
* fix TestSysUnseal_Reset
* attempt to add build-date to release builds
* add branch to github build workflow
* add get-build-date to build-* job needs
* fix release build command vars
* add missing quote in release build command
* Revert "add branch to github build workflow"
This reverts commit b835699ecb7c2c632757fa5fe64b3d5f60d2a886.
* add changelog entry
Use the latest version of the actions-packaging-linux@v1 to ensure that
.deb and .rpm artifacts are generated with release.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* achieve parity with ent in core.go
* add VAULT_DISABLE_LOCAL_AUTH_MOUNT_ENTITIES
* parity in build.yml with ent but without adding the +ent
* pass base version to ldflags
Co-authored-by: Kyle Penfound <kpenfound11@gmail.com>
* Make backport assistant work with ENT backport tags
I want the regexp to include word characters and the plus sign on
enterprise. I'm confused about the levels of escaping required, because
this is a shell env var that gets passed to a go program as a regular
expression. I didn't escape the square brackets because the parens
weren't escaped. I matched what was done previously with the '.'s for
the '+' because I also want that literal, but it was unclear from
playing around with https://regex101.com/ and go regexps whether the
extra backslashes were required for the '+' char.
* Update .github/workflows/backport.yml
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
This way we can have an OSS action, and an ENT action. We'll be able to
keep the backport-assistant workflows the same on OSS and ENT for normal
PRs, and have a separate labeling system on OSS for docs backports.
We should delete the backport-docs.yml file on ENT.
* feat: add `backport.yml`
* feat: check releases over labels for latest
* chore: Do not automerge for non `stable-website`
* chore: Do not automerge to not `stable-website`
* Update .github/workflows/backport.yml
Co-authored-by: Meggie <meggie@hashicorp.com>
Co-authored-by: Meggie <meggie@hashicorp.com>
* adding CRT to main branch
* cleanup
* um i dont know how that got removed but heres the fix
* add vault.service
Co-authored-by: Kyle Penfound <kpenfound11@gmail.com>
* Adding github action to check for changelog file
This might have to be slightly different on ENT, where changelog files
should be prefixed with an underscore.
* Fixing comment
* Adding release branches, enhancement more specific
Also tidying my rusty bash
* More descriptive error messages
I also un-nested some if statements.
* Reference this PR too
* website: bump to docs-page prerelease, support hidden pages
* website: switch to hidden pages approach for docs and api-docs
* website: remove temporary fix for hidden pages, and related check
* website: fix content structure issue with docs/auth/jwt
* website: bump to latest pre-release
* website: bump to stable docs-page, w next-mdx-remote bump
* website: bump to latest markdown-page
