scottk/vault
1
0
Fork 0
mirror of https://github.com/optim-enterprises-bv/vault.git synced 2025-10-29 09:42:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
9a10689ca3a3ccadb24453c801b715dd11376d9a
vault/Makefile
Ryan Cragun 9a10689ca3 [QT-645] Restructure dev tools (#24559) 
We're on a quest to reduce our pipeline execution time to both enhance
our developer productivity but also to reduce the overall cost of the CI
pipeline. The strategy we use here reduces workflow execution time and
network I/O cost by reducing our module cache size and using binary
external tools when possible. We no longer download modules and build
many of the external tools thousands of times a day.

Our previous process of installing internal and external developer tools
was scattered and inconsistent. Some tools were installed via `go
generate -tags tools ./tools/...`,
others via various `make` targets, and some only in Github Actions
workflows. This process led to some undesirable side effects:
  * The modules of some dev and test tools were included with those
    of the Vault project. This leads to us having to manage our own
    Go modules with those of external tools. Prior to Go 1.16 this
    was the recommended way to handle external tools, but now
    `go install tool@version` is the recommended way to handle
    external tools that need to be build from source as it supports
    specific versions but does not modify the go.mod.
  * Due to Github cache constraints we combine our build and test Go
    module caches together, but having our developer tools as deps in
    our module results in a larger cache which is downloaded on every
    build and test workflow runner. Removing the external tools that were
    included in our go.mod reduced the expanded module cache by size
    by ~300MB, thus saving time and network I/O costs when downloading
    the module cache.
  * Not all of our developer tools were included in our modules. Some were
    being installed with `go install` or `go run`, so they didn't take
    advantage of a single module cache. This resulted in us downloading
    Go modules on every CI and Build runner in order to build our
    external tools.
  * Building our developer tools from source in CI is slow. Where possible
    we can prefer to use pre-built binaries in CI workflows. No more
    module download or tool compiles if we can avoid them.

I've refactored how we define internal and external build tools
in our Makefile and added several new targets to handle both building
the developer tools locally for development and verifying that they are
available. This allows for an easy developer bootstrap while also
supporting installation of many of the external developer tools from
pre-build binaries in CI. This reduces our network IO and run time
across nearly all of our actions runners.

While working on this I caught and resolved a few unrelated issue:
* Both our Go and Proto format checks we're being run incorrectly. In
  CI they we're writing changes but not failing if changes were
  detected. The Go was less of a problem as we have git hooks that
  are intended to enforce formatting, however we drifted over time.
* Our Git hooks couldn't handle removing a Go file without failing. I
  moved the diff check into the new Go helper and updated it to handle
  removing files.
* I combined a few separate scripts and into helpers and added a few
  new capabilities.
* I refactored how we install Go modules to make it easier to download
  and tidy all of the projects go.mod's.
* Refactor our internal and external tool installation and verification
  into a tools.sh helper.
* Combined more complex Go verification into `scripts/go-helper.sh` and
  utilize it in the `Makefile` and git commit hooks.
* Add `Makefile` targets for executing our various tools.sh helpers.
* Update our existing `make` targets to use new tool targets.
* Normalize our various scripts and targets output to have a consistent
  output format.
* In CI, install many of our external dependencies as binaries wherever
  possible. When not possible we'll build them from scratch but not mess
  with the shared module cache.
* [QT-641] Remove our external build tools from our project Go modules.
* [QT-641] Remove extraneous `go list`'s from our `set-up-to` composite
  action.
* Fix formatting and regen our protos

Signed-off-by: Ryan Cragun <me@ryan.ec>
2024-01-09 17:50:46 +00:00

372 lines
14 KiB
Makefile
Raw Blame History

# Determine this makefile's path.
# Be sure to place this BEFORE `include` directives, if any.
THIS_FILE := $(lastword $(MAKEFILE_LIST))
TEST?=$$($(GO_CMD) list ./... | grep -v /vendor/ | grep -v /integ)
TEST_TIMEOUT?=45m
EXTENDED_TEST_TIMEOUT=60m
INTEG_TEST_TIMEOUT=120m
VETARGS?=-asmdecl -atomic -bool -buildtags -copylocks -methods -nilfunc -printf -rangeloops -shift -structtags -unsafeptr
GOFMT_FILES?=$$(find . -name '*.go' | grep -v pb.go | grep -v vendor)
SED?=$(shell command -v gsed || command -v sed)
GO_VERSION_MIN=$$(cat $(CURDIR)/.go-version)
GO_CMD?=go
CGO_ENABLED?=0
ifneq ($(FDB_ENABLED), )
CGO_ENABLED=1
BUILD_TAGS+=foundationdb
endif
default: dev
# bin generates the releasable binaries for Vault
bin: prep
@CGO_ENABLED=$(CGO_ENABLED) BUILD_TAGS='$(BUILD_TAGS) ui' sh -c "'$(CURDIR)/scripts/build.sh'"
# dev creates binaries for testing Vault locally. These are put
# into ./bin/ as well as $GOPATH/bin
dev: BUILD_TAGS+=testonly
dev: prep
@CGO_ENABLED=$(CGO_ENABLED) BUILD_TAGS='$(BUILD_TAGS)' VAULT_DEV_BUILD=1 sh -c "'$(CURDIR)/scripts/build.sh'"
dev-ui: BUILD_TAGS+=testonly
dev-ui: assetcheck prep
@CGO_ENABLED=$(CGO_ENABLED) BUILD_TAGS='$(BUILD_TAGS) ui' VAULT_DEV_BUILD=1 sh -c "'$(CURDIR)/scripts/build.sh'"
dev-dynamic: BUILD_TAGS+=testonly
dev-dynamic: prep
@CGO_ENABLED=1 BUILD_TAGS='$(BUILD_TAGS)' VAULT_DEV_BUILD=1 sh -c "'$(CURDIR)/scripts/build.sh'"
# *-mem variants will enable memory profiling which will write snapshots of heap usage
# to $TMP/vaultprof every 5 minutes. These can be analyzed using `$ go tool pprof <profile_file>`.
# Note that any build can have profiling added via: `$ BUILD_TAGS=memprofiler make ...`
dev-mem: BUILD_TAGS+=memprofiler
dev-mem: dev
dev-ui-mem: BUILD_TAGS+=memprofiler
dev-ui-mem: assetcheck dev-ui
dev-dynamic-mem: BUILD_TAGS+=memprofiler
dev-dynamic-mem: dev-dynamic
# Creates a Docker image by adding the compiled linux/amd64 binary found in ./bin.
# The resulting image is tagged "vault:dev".
docker-dev: BUILD_TAGS+=testonly
docker-dev: prep
docker build --build-arg VERSION=$(GO_VERSION_MIN) --build-arg BUILD_TAGS="$(BUILD_TAGS)" -f scripts/docker/Dockerfile -t vault:dev .
docker-dev-ui: BUILD_TAGS+=testonly
docker-dev-ui: prep
docker build --build-arg VERSION=$(GO_VERSION_MIN) --build-arg BUILD_TAGS="$(BUILD_TAGS)" -f scripts/docker/Dockerfile.ui -t vault:dev-ui .
# test runs the unit tests and vets the code
test: BUILD_TAGS+=testonly
test: prep
@CGO_ENABLED=$(CGO_ENABLED) \
VAULT_ADDR= \
VAULT_TOKEN= \
VAULT_DEV_ROOT_TOKEN_ID= \
VAULT_ACC= \
$(GO_CMD) test -tags='$(BUILD_TAGS)' $(TEST) $(TESTARGS) -timeout=$(TEST_TIMEOUT) -parallel=20
testcompile: BUILD_TAGS+=testonly
testcompile: prep
@for pkg in $(TEST) ; do \
$(GO_CMD) test -v -c -tags='$(BUILD_TAGS)' $$pkg -parallel=4 ; \
done
# testacc runs acceptance tests
testacc: BUILD_TAGS+=testonly
testacc: prep
@if [ "$(TEST)" = "./..." ]; then \
echo "ERROR: Set TEST to a specific package"; \
exit 1; \
fi
VAULT_ACC=1 $(GO_CMD) test -tags='$(BUILD_TAGS)' $(TEST) -v $(TESTARGS) -timeout=$(EXTENDED_TEST_TIMEOUT)
# testrace runs the race checker
testrace: BUILD_TAGS+=testonly
testrace: prep
@CGO_ENABLED=1 \
VAULT_ADDR= \
VAULT_TOKEN= \
VAULT_DEV_ROOT_TOKEN_ID= \
VAULT_ACC= \
$(GO_CMD) test -tags='$(BUILD_TAGS)' -race $(TEST) $(TESTARGS) -timeout=$(EXTENDED_TEST_TIMEOUT) -parallel=20
cover:
./scripts/coverage.sh --html
# vet runs the Go source code static analysis tool `vet` to find
# any common errors.
vet:
@$(GO_CMD) list -f '{{.Dir}}' ./... | grep -v /vendor/ \
| grep -v '.*github.com/hashicorp/vault$$' \
| xargs $(GO_CMD) vet ; if [ $$? -eq 1 ]; then \
echo ""; \
echo "Vet found suspicious constructs. Please check the reported constructs"; \
echo "and fix them if necessary before submitting the code for reviewal."; \
fi
# deprecations runs staticcheck tool to look for deprecations. Checks entire code to see if it
# has deprecated function, variable, constant or field
deprecations: bootstrap prep
@BUILD_TAGS='$(BUILD_TAGS)' ./scripts/deprecations-checker.sh ""
# ci-deprecations runs staticcheck tool to look for deprecations. All output gets piped to revgrep
# which will only return an error if changes that is not on main has deprecated function, variable, constant or field
ci-deprecations: prep check-tools-external
@BUILD_TAGS='$(BUILD_TAGS)' ./scripts/deprecations-checker.sh main
# vet-codechecker runs our custom linters on the test functions. All output gets
# piped to revgrep which will only return an error if new piece of code violates
# the check
vet-codechecker: check-tools-internal
@echo "==> Running go vet with ./tools/codechecker..."
@$(GO_CMD) vet -vettool=$$(which codechecker) -tags=$(BUILD_TAGS) ./... 2>&1 | revgrep
# vet-codechecker runs our custom linters on the test functions. All output gets
# piped to revgrep which will only return an error if new piece of code that is
# not on main violates the check
ci-vet-codechecker: tools-internal check-tools-external
@echo "==> Running go vet with ./tools/codechecker..."
@$(GO_CMD) vet -vettool=$$(which codechecker) -tags=$(BUILD_TAGS) ./... 2>&1 | revgrep origin/main
# lint runs vet plus a number of other checkers, it is more comprehensive, but louder
lint: check-tools-external
@$(GO_CMD) list -f '{{.Dir}}' ./... | grep -v /vendor/ \
| xargs golangci-lint run; if [ $$? -eq 1 ]; then \
echo ""; \
echo "Lint found suspicious constructs. Please check the reported constructs"; \
echo "and fix them if necessary before submitting the code for reviewal."; \
fi
# for ci jobs, runs lint against the changed packages in the commit
ci-lint: check-tools-external
@golangci-lint run --deadline 10m --new-from-rev=HEAD~
# Lint protobuf files
protolint: prep check-tools-external
@echo "==> Linting protobufs..."
@buf lint
# prep runs `go generate` to build the dynamically generated
# source files.
#
# n.b.: prep used to depend on fmtcheck, but since fmtcheck is
# now run as a pre-commit hook (and there's little value in
# making every build run the formatter), we've removed that
# dependency.
prep: check-go-version
@echo "==> Running go generate..."
@GOARCH= GOOS= $(GO_CMD) generate $$($(GO_CMD) list ./... | grep -v /vendor/)
@if [ -d .git/hooks ]; then cp .hooks/* .git/hooks/; fi
# bootstrap the build by generating any necessary code and downloading additional tools that may
# be used by devs.
bootstrap: prep tools
# Note: if you have plugins in GOPATH you can update all of them via something like:
# for i in $(ls | grep vault-plugin-); do cd $i; git remote update; git reset --hard origin/master; dep ensure -update; git add .; git commit; git push; cd ..; done
update-plugins:
grep vault-plugin- go.mod | cut -d ' ' -f 1 | while read -r P; do echo "Updating $P..."; go get -v "$P"; done
static-assets-dir:
@mkdir -p ./http/web_ui
install-ui-dependencies:
@echo "==> Installing JavaScript assets"
@cd ui && yarn
test-ember: install-ui-dependencies
@echo "==> Running ember tests"
@cd ui && yarn run test:oss
test-ember-enos: install-ui-dependencies
@echo "==> Running ember tests with a real backend"
@cd ui && yarn run test:enos
ember-dist: install-ui-dependencies
@cd ui && npm rebuild node-sass
@echo "==> Building Ember application"
@cd ui && yarn run build
@rm -rf ui/if-you-need-to-delete-this-open-an-issue-async-disk-cache
ember-dist-dev: install-ui-dependencies
@cd ui && npm rebuild node-sass
@echo "==> Building Ember application"
@cd ui && yarn run build:dev
static-dist: ember-dist
static-dist-dev: ember-dist-dev
proto: check-tools-external
@echo "==> Generating Go code from protobufs..."
buf generate
# No additional sed expressions should be added to this list. Going forward
# we should just use the variable names choosen by protobuf. These are left
# here for backwards compatability, namely for SDK compilation.
$(SED) -i -e 's/Id/ID/' -e 's/SPDX-License-IDentifier/SPDX-License-Identifier/' vault/request_forwarding_service.pb.go
$(SED) -i -e 's/Idp/IDP/' -e 's/Url/URL/' -e 's/Id/ID/' -e 's/IDentity/Identity/' -e 's/EntityId/EntityID/' -e 's/Api/API/' -e 's/Qr/QR/' -e 's/Totp/TOTP/' -e 's/Mfa/MFA/' -e 's/Pingid/PingID/' -e 's/namespaceId/namespaceID/' -e 's/Ttl/TTL/' -e 's/BoundCidrs/BoundCIDRs/' -e 's/SPDX-License-IDentifier/SPDX-License-Identifier/' helper/identity/types.pb.go helper/identity/mfa/types.pb.go helper/storagepacker/types.pb.go sdk/plugin/pb/backend.pb.go sdk/logical/identity.pb.go vault/activity/activity_log.pb.go
# This will inject the sentinel struct tags as decorated in the proto files.
protoc-go-inject-tag -input=./helper/identity/types.pb.go
protoc-go-inject-tag -input=./helper/identity/mfa/types.pb.go
fmt:
find . -name '*.go' | grep -v pb.go | grep -v vendor | xargs gofumpt -w
fmtcheck: check-go-fmt
.PHONY: go-mod-download
go-mod-download:
@$(CURDIR)/scripts/go-helper.sh mod-download
.PHONY: go-mod-tidy
go-mod-tidy:
@$(CURDIR)/scripts/go-helper.sh mod-tidy
protofmt:
buf format -w
semgrep:
semgrep --include '*.go' --exclude 'vendor' -a -f tools/semgrep .
assetcheck:
@echo "==> Checking compiled UI assets..."
@sh -c "'$(CURDIR)/scripts/assetcheck.sh'"
spellcheck:
@echo "==> Spell checking website..."
@misspell -error -source=text website/source
.PHONY check-go-fmt:
check-go-fmt:
@$(CURDIR)/scripts/go-helper.sh check-fmt
.PHONY check-go-version:
check-go-version:
@$(CURDIR)/scripts/go-helper.sh check-version $(GO_VERSION_MIN)
.PHONY: check-proto-fmt
check-proto-fmt:
buf format -d --error-format github-actions --exit-code
.PHONY: check-proto-delta
check-proto-delta: prep
@echo "==> Checking for a delta in proto generated Go files..."
@echo "==> Deleting all *.pg.go files..."
find . -type f -name '*.pb.go' -delete -print0
@$(MAKE) -f $(THIS_FILE) proto
@if ! git diff --exit-code; then echo "Go protobuf bindings need to be regenerated. Run 'make proto' to fix them." && exit 1; fi
.PHONY:check-sempgrep
check-sempgrep: check-tools-external
@echo "==> Checking semgrep..."
@semgrep --error --include '*.go' --exclude 'vendor' -f tools/semgrep/ci .
.PHONY: check-tools
check-tools:
@$(CURDIR)/tools/tools.sh check
.PHONY: check-tools-external
check-tools-external:
@$(CURDIR)/tools/tools.sh check-external
.PHONY: check-tools-internal
check-tools-internal:
@$(CURDIR)/tools/tools.sh check-internal
check-vault-in-path:
@VAULT_BIN=$$(command -v vault) || { echo "vault command not found"; exit 1; }; \
[ -x "$$VAULT_BIN" ] || { echo "$$VAULT_BIN not executable"; exit 1; }; \
printf "Using Vault at %s:\n\$$ vault version\n%s\n" "$$VAULT_BIN" "$$(vault version)"
.PHONY: tools
tools:
@$(CURDIR)/tools/tools.sh install
.PHONY: tools-external
tools-external:
@$(CURDIR)/tools/tools.sh install-external
.PHONY: tools-internal
tools-internal:
@$(CURDIR)/tools/tools.sh install-internal
mysql-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/mysql-database-plugin ./plugins/database/mysql/mysql-database-plugin
mysql-legacy-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/mysql-legacy-database-plugin ./plugins/database/mysql/mysql-legacy-database-plugin
cassandra-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/cassandra-database-plugin ./plugins/database/cassandra/cassandra-database-plugin
influxdb-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/influxdb-database-plugin ./plugins/database/influxdb/influxdb-database-plugin
postgresql-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/postgresql-database-plugin ./plugins/database/postgresql/postgresql-database-plugin
mssql-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/mssql-database-plugin ./plugins/database/mssql/mssql-database-plugin
hana-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/hana-database-plugin ./plugins/database/hana/hana-database-plugin
mongodb-database-plugin:
@CGO_ENABLED=0 $(GO_CMD) build -o bin/mongodb-database-plugin ./plugins/database/mongodb/mongodb-database-plugin
# These ci targets are used for used for building and testing in Github Actions
# workflows and for Enos scenarios.
.PHONY: ci-build
ci-build:
@$(CURDIR)/scripts/ci-helper.sh build
.PHONY: ci-build-ui
ci-build-ui:
@$(CURDIR)/scripts/ci-helper.sh build-ui
.PHONY: ci-bundle
ci-bundle:
@$(CURDIR)/scripts/ci-helper.sh bundle
.PHONY: ci-get-artifact-basename
ci-get-artifact-basename:
@$(CURDIR)/scripts/ci-helper.sh artifact-basename
.PHONY: ci-get-date
ci-get-date:
@$(CURDIR)/scripts/ci-helper.sh date
.PHONY: ci-get-revision
ci-get-revision:
@$(CURDIR)/scripts/ci-helper.sh revision
.PHONY: ci-get-version-package
ci-get-version-package:
@$(CURDIR)/scripts/ci-helper.sh version-package
.PHONY: ci-install-external-tools
ci-install-external-tools:
@$(CURDIR)/scripts/ci-helper.sh install-external-tools
.PHONY: ci-prepare-legal
ci-prepare-legal:
@$(CURDIR)/scripts/ci-helper.sh prepare-legal
.PHONY: ci-update-external-tool-modules
ci-update-external-tool-modules:
@$(CURDIR)/scripts/ci-helper.sh update-external-tool-modules
.PHONY: ci-copywriteheaders
ci-copywriteheaders:
copywrite headers --plan
# Special case for MPL headers in /api, /sdk, and /shamir
cd api && $(CURDIR)/scripts/copywrite-exceptions.sh
cd sdk && $(CURDIR)/scripts/copywrite-exceptions.sh
cd shamir && $(CURDIR)/scripts/copywrite-exceptions.sh
.PHONY: all bin default prep test vet bootstrap fmt fmtcheck mysql-database-plugin mysql-legacy-database-plugin cassandra-database-plugin influxdb-database-plugin postgresql-database-plugin mssql-database-plugin hana-database-plugin mongodb-database-plugin ember-dist ember-dist-dev static-dist static-dist-dev assetcheck check-vault-in-path packages build build-ci semgrep semgrep-ci vet-codechecker ci-vet-codechecker clean dev
.NOTPARALLEL: ember-dist ember-dist-dev
Reference in New Issue View Git Blame Copy Permalink