* Update the if conditions for test-go jobs in CI
* Fix errors in comments
* Update the if condition for the UI tests
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* backport of commit daf4ec2b4f
* Add compute-huge output to setup step in CI
* Remove redundant fromJSON call
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* backport all gha migration changes to release/1.13.x
* remove the .circleci directory
* remove references to circleci configuration from pre-commit hook
* remove reference to .circleci in Makefile
* port change to how gofumpt is executed in Makefile
* add gotestsum to tools/tools.go
* remove postgresql/scram package from generate-test-package-lists.sh since it didn't exist in release 1.13 or earlier
* blank out environment variables to allow test to properly function
* use go:embed to load files into test
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* example for checking go doc tests
* add analyzer test and action
* get metadata step
* install revgrep
* fix for ci
* add revgrep to go.mod
* clarify how analysistest works
This uses aws-nuke and awslimitchecker to monitor the new vault CI account to clean up and prevent resource quota exhaustion. AWS-nuke will scan all regions of the accounts for lingering resources enos/terraform didn't clean up, and if they don't match exclusion criteria, delete them every night. By default, we exclude corp-sec created resources, our own CI resources, and when possible, anything created within the past 72 hours. Because this account is dedicated to CI, users should not expect resources to persist beyond this without additional configuration.
- enos-run workflow will checkout the `main` branch by default, which would pass incorrect metadata to the workflow
so we use the `revision` passed by the calling workflow to checkout the sha and get the relavant metadata`
Signed-off-by: Jaymala Sinha <jaymala@hashicorp.com>
Signed-off-by: Jaymala Sinha <jaymala@hashicorp.com>
Introducing a new approach to testing Vault artifacts before merge
and after merge/notorization/signing. Rather than run a few static
scenarios across the artifacts, we now have the ability to run a
pseudo random sample of scenarios across many different build artifacts.
We've added 20 possible scenarios for the AMD64 and ARM64 binary
bundles, which we've broken into five test groups. On any given push to
a pull request branch, we will now choose a random test group and
execute its corresponding scenarios against the resulting build
artifacts. This gives us greater test coverage but lets us split the
verification across many different pull requests.
The post-merge release testing pipeline behaves in a similar fashion,
however, the artifacts that we use for testing have been notarized and
signed prior to testing. We've also reduce the number of groups so that
we run more scenarios after merge to a release branch.
We intend to take what we've learned building this in Github Actions and
roll it into an easier to use feature that is native to Enos. Until then,
we'll have to manually add scenarios to each matrix file and manually
number the test group. It's important to note that Github requires every
matrix to include at least one vector, so every artifact that is being
tested must include a single scenario in order for all workflows to pass
and thus satisfy branch merge requirements.
* Add support for different artifact types to enos-run
* Add support for different runner type to enos-run
* Add arm64 scenarios to build matrix
* Expand build matrices to include different variants
* Update Consul versions in Enos scenarios and matrices
* Refactor enos-run environment
* Add minimum version filtering support to enos-run. This allows us to
automatically exclude scenarios that require a more recent version of
Vault
* Add maximum version filtering support to enos-run. This allows us to
automatically exclude scenarios that require an older version of
Vault
* Fix Node 12 deprecation warnings
* Rename enos-verify-stable to enos-release-testing-oss
* Convert artifactory matrix into enos-release-testing-oss matrices
* Add all Vault editions to Enos scenario matrices
* Fix verify version with complex Vault edition metadata
* Rename the crt-builder to ci-helper
* Add more version helpers to ci-helper and Makefile
* Update CODEOWNERS for quality team
* Add support for filtering matrices by group and version constraints
* Add support for pseudo random test scenario execution
Signed-off-by: Ryan Cragun <me@ryan.ec>
