* Fix the build notification. It appears that during a rebase the JSON
payload was slightly corrupted.
* Don't create a successful CI step summary if the CI workflow is
cancelled.
* Don't create a successful CI comment if the workflow was cancelled.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Context
-------
Building and testing Vault artifacts on pull requests and merges is
responsible for about 1/3rd of our overall spend on Vault CI. Of the
artifacts that we ship as part of a release, we do Enos testing scenarios
on the `linux/amd64` and `linux/arm64` binaries and their derivative
artifacts. The extended build artifacts for non-Linux platforms or less
common machine architectures are not tested at this time. They are built,
notarized, and signed as part of every pull request update and merge. As
we don't actually test these artifacts, the only gain we get from this
rather expensive behavior is that we wont merge a change that would prevent
Vault from building on one of the extended targets. Extended platform or
architecture changes are quite rare, so performing this work as frequently
as we do is costly in both monetary and developer time for little relative
safety benefit.
Goals
-----
Rethink and implement how and when we build binaries and artifacts of Vault
so that we can spend less money on repetitive work and while also reducing
the time it takes for the build and test pipelines to complete.
Solution
--------
Instead of building all release artifacts on every push, we'll opt to build
only our testable (core) artifacts. With this change we are introducing a
bit of risk. We could merge a change that breaks an extended platform and
only find out after the fact when we trigger a complete build for a release.
We'll hedge against that risk by building all of the release targets on a
scheduled cadence to ensure that they are still buildable.
We'll make building all of the targets optional on any pull request by
use of a `build/all` label on the pull request.
Further considerations
----------------------
* We want to reduce the total number of workflows and runners for all of our
pipelines if possible. As each workflow runner has infrastructure cost and
runner time penalties, using a single runner over many is often preferred.
* Many of our jobs runners have been optimized for cost and performance. We
should simplify the choices of which runners to use.
* CRT requires us to use the same build workflow in both CE and Ent.
Historically that meant that modifying `build.yml` in CE would result in a
merge conflict with `build.yml` in Ent, and break our merge workflows.
* Workflow flow control in both `build.yml` and `ci.yml` can be quite
complicated, as each needs to maintain compatibility whether executed as CE
or Ent, and when triggered with various Github events like pull_request,
push, and workflow_call, each with their own requirements.
* Many jobs utilize similar patterns of flow control and metadata but are not
reusable.
* Workflow call depth has a maximum of four, so we need to be quite
considerate when calling other workflows.
* Called workflows can only have 10 inputs.
Implementation
--------------
* Refactor the `build.yml` workflow to be agnostic to whether or not it is
executing in CE or Ent. That makes future updates to the build much easier
as we won't have to worry about merge conflicts when the change is merged
downstream.
* Extract common steps in workflows into composite actions that we can reuse.
* Fix bugs where some but not all workflows would use different Git
references when building and testing a pull request.
* We rewrite the application, docs, and UI change helpers as a composite
action. This allows us to re-use this logic to make consistent behavior
choices across build and CI.
* We combine several `build.yml` and `ci.yml` jobs into our final job.
This reduces the number of workflows required for the same behavior while
saving time overall.
* Update most of our action pins.
Results
-------
| Metric | Before | After | Diff |
|-------------------|----------|---------|-------|
| Duration: | ~14-18m | ~15-18m | ~ = |
| Workflows: | 43 | 18 | - 58% |
| Billable time: | ~1h15m | 16m | - 79% |
| Saved artifacts: | 34 | 12 | - 65% |
Infra costs should map closely to billable time.
Network I/O costs should map closely to the workflow count.
Storage costs should map directly with saved artifacts.
We could probably get parity with duration by getting more clever with
our UBI container build, as that's where we're seeing the increase. I'm
not yet concerned as it takes roughly the same time for this job to
complete as it did before.
While the CI workflow was not the focus on the PR, some shared
refactoring does show some marginal improvements there.
| Metric | Before | After | Diff |
|-------------------|----------|----------|--------|
| Duration: | ~24m | ~12.75m | - 15% |
| Workflows: | 55 | 47 | - 8% |
| Billable time: | ~4h20m | ~3h36m | - 7% |
Further focus on streamlining the CI workflows would likely result in a
few more marginal improvements, but nothing on the order like we've seen
with the build workflow.
Signed-off-by: Ryan Cragun <me@ryan.ec>
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License.
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* add missing license headers
* Update copyright file headers to BUS-1.1
* Fix test that expected exact offset on hcl file
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* limit test comments
* remove unecessary tee
* fix go test condition
* fix
* fail test
* remove ailways entirely
* fix columns
* make a bunch of tests fail
* separate line
* include Failures:
* remove test fails
* fix whitespace
* fix multiline
* shellcheck, and success message for builds
* add full path
* cat the summary
* fix and faster
* fix if condition
* base64 in a separate step
* echo
* check against empty string
* add echo
* only use matrix ids
* only id
* echo matrix
* remove wrapping array
* tojson
* try echo again
* use jq to get packages
* don't quote
* only run binary tests once
* only run binary tests once
* test what's wrong with the binary
* separate file
* use matrix file
* failed test
* update comment on success
* correct variable name
* bae64 fix
* output to file
* use multiline
* fix
* fix formatting
* fix newline
* fix whitespace
* correct body, remove comma
* small fixes
* shellcheck
* another shellcheck fix
* fix deprecation checker
* only run comments for prs
* Update .github/workflows/test-go.yml
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* Update .github/workflows/test-go.yml
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* fixes
---------
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* report build failures in a PR comment
* address action linter
* linter
* add an id
* change permission
* report failure from build yaml
* linter fix
* report workflow url
* reorder jobs
* complete boolean eval
* single quote
* experiment getting failed jobs
* linter
* pass failed jobs one by one
* failed jobs are reported cancelled
* use * instead of @
* some polishing
* find comment ID, create or update it
* some clean up
* missing }
Integrate the `test-go` workflow with `gotestsum tool ci-matrix`. The
tool uses the output of `go list ./...` along with timing files emitted
by `gotestsum` to generate a test matrix of 16 runners with evenly
distributed runtimes.
We intentionally ignore binary, docker-based test files for the initial
matrix creation and then inject a 17th runner, dedicated to building
Vault and running the entire binary test suite together. This avoids
duplication of build overhead when binary tests are rebalanced across
multiple runners in the generated matrix.
In order to maintain test results from previous runs, we cache the test
results after every run of `gotestsum`. Each cache entry occupies ~36MB
after compression on enterprise, at the time of this commit.
We'll have to keep an eye on this to make sure timing data is not
evicted from the cache, but in theory it should be toward the top of the
LRU entries.
* use verify changes for docs to skip tests
* add verify-changes to the needed jobs
* skip go tests for doc/ui only changes
* fix a job ref
* change names, remove script
* remove ui conditions
* separate flags
* feedback
* combine into one checker
* combine and simplify ci checks
* add to test package list
* remove testing test
* only run deprecations check
* only run deprecations check
* remove unneeded repo check
* fix bash options
The upcoming event main plugin will use a very similar pattern
as the database plugin map, so it makes sense to refactor this and move
this map out. It also cleans up the database plugin backend so that
it does not have to keep track of the lock.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* VAULT-15385 Add GHA that checks for nil, nil returns on functions that return an error
* VAULT-15385 add failing function, for sanity
* VAULT-15385 fix makefile
* VAULT-15385 remove test dir
* VAULT-15385 Fix typo
* VAULT-15385 fix job name
* VAULT-15385 Add test to packages
* VAULT-15835 add opt-out
* VAULT-15835 Wrong file for comment
* VAULT-15835 remove failing function
* VAULT-15835 return not nil-nil :)
* VAULT-15835 Restrict to two-result functions
* VAULT-15547 First pass at agent/proxy decoupling
* VAULT-15547 Fix some imports
* VAULT-15547 cases instead of string.Title
* VAULT-15547 changelog
* VAULT-15547 Fix some imports
* VAULT-15547 some more dependency updates
* VAULT-15547 More dependency paths
* VAULT-15547 godocs for tests
* VAULT-15547 godocs for tests
* VAULT-15547 test package updates
* VAULT-15547 test packages
* VAULT-15547 add proxy to test packages
* VAULT-15547 gitignore
* VAULT-15547 address comments
* VAULT-15547 Some typos and small fixes
* Update verify-changes to support external docs branches
Signed-off-by: Jaymala Sinha <jaymala@hashicorp.com>
* Revert QT-545 as it Enos workflow is not a workflow_run event
Signed-off-by: Jaymala Sinha <jaymala@hashicorp.com>
---------
Signed-off-by: Jaymala Sinha <jaymala@hashicorp.com>
* Migrate subset of CircleCI ci workflow to GitHub Actions
Runs test-go and test-go-remote-docker with a static splitting of test packages
* [skip actions] add comment to explain the purpose of test-generate-test-package-lists.sh and what to do if it fails
* change trigger to push
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
