mirror of
https://github.com/optim-enterprises-bv/vault.git
synced 2025-12-26 15:27:26 +00:00
29cc2b2586
We further optimize the CI workflow for better costs and speed. We tested the Go CI workflows across several instance classes and update our compute choices. We achieve an average execution speed improvement of 2-2.5 minutes per test workflow while reducing the infrastructure cost by about 20%. We also also save another ~2 minutes by installing `gotestsum` from the Github release instead of downloading the Go modules and compiling it every time. In addition to the speed improvements, we also further reduced our cache usage by updating the `security-scan` workflow to not cache Go modules. We also use the `cache/save` and `cache/restore` actions for timing caches. This results is saving half as many cache results for timing data. *UI test results* results for 2x runs: * c6a.2xlarge (12m54s, 11m55s) * c6a.4xlarge (10m47s, 11m6s) * c6a.8xlarge (11m32s, 10m51s) * m5.2xlarge (15m23s, 14m16s) * m5.4xlarge (14m48s, 12m54s) * m5.8xlarge (12m27s, 12m24s) * m6a.2xlarge (11m55s, 12m20s) * m6a.4xlarge (10m54s, 10m43s) * m6a.8xlarge (10m33s, 10m51s) Current runner: m5.2xlarge (15m23s, 14m16s, avg 14m50s) @ 0.448/hr = $0.11 Faster candidates * c6a.2xlarge (12m54s, 11m55s, avg 12m24s) @ 0.3816/hr = $0.078 * m6a.2xlarge (11m55s, 12m20s, avg 12m8s) @ 0.4032/hr = $0.081 * c6a.4xlarge (10m47s, 11m6s, avg 10m56s) @ 0.7632/hr = $0.139 * m6a.4xlarge (10m54s, 10m43s, avg 10m48s) @ 0.8064/hr = $0.140 Best bang for the buck for test-ui: m6a.2xlarge, > 25% cost savings from current and we save ~2.5 minutes. *Go test results* During testing the external replication tests, when not broken up, will always take the longest. Our original analysis focuses on this job. Most other tests groups will finish ~3m faster so we'll use subtract that time when estimating the cost for the whole job. external replication job results: * c6a.2xlarge (20m49s, 19m20s, avg 20m5s) * c6a.4xlarge (19m1s, 19m38s, avg 19m20s) * c6a.8xlarge (19m51s, 18m54s, avg 19m23s) * m5.2xlarge (22m12s, 20m29s, avg 21m20s) * m5.4xlarge (20m7s, 19m3s, avg 20m35s) * m5.8xlarge (20m24s, 19m42s, avg 20m3s) * m6a.2xlarge (21m10s, 19m37s, avg 20m23s) * m6a.4xlarge (18m58s, 19m51s, avg 19m24s) * m6a.8xlarge (19m27s, 18m47s, avg 19m7s) There is little separation in time when we increase class size. In the best case a class size increase yields about a ~5% performance increase and doubles the cost. For test-go our best bang for the buck is certainly going to be in the 2xlarge class. Current runner: m5.2xlarge (22m12s, 20m29s, avg 21m20s) @ 0.448/hr (16@avg-3m + 1@avg) = $2.35 Candidates in the same class * c6a.2xlarge (20m49s, 19m20s, avg 20m5s) @ 0.3816/hr (16@avg-3m + 1@avg) = $1.86 * m6a.2xlarge (21m10s, 19m37s, avg 20m23s) @ 0.4032/hr (16@avg-3m + 1@avg) = $2.00 Best bang for the buck for test-go: c6a.2xlarge: 20% cost savings and save about ~2.25 minutes. We ran the tests with similar instances and saw similar execution times as with test-go. Therefore we can use the same recommended instance sizes. After breaking up test-go's external replication tests, the longest group was shorter on average. I choose to look at group 3 as it was usually the longest grouping: * c6a.2xlarge: (14m51s, 14m48s) * c6a.4xlarge: (14m14s, 14m15) * c6a.8xlarge: (14m0s, 13m54s) * m5.2xlarge: (15m36s, 15m35s) * m5.4xlarge: (14m46s, 14m49s) * m5.8xlarge: (14m25s, 14m25s) * m6a.2xlarge: 14m51s, 14m53s) * m6a.4xlarge: 14m16s, 14m16s) * m6a.8xlarge: (14m2s, 13m57s) Again, we see ~5% performance gains between the 2x and 8x instance classes at quadruple the cost. The c6a and m6a families are almost identical, with the c6a class being cheaper. *Notes* * UI and Go Test timing results: https://github.com/hashicorp/vault-enterprise/actions/runs/5556957460/jobs/10150759959 * Go Test with data race detection timing results: https://github.com/hashicorp/vault-enterprise/actions/runs/5558013192 * Go Test with replication broken up: https://github.com/hashicorp/vault-enterprise/actions/runs/5558490899 Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Ryan Cragun <me@ryan.ec>
86 lines
2.7 KiB
YAML
86 lines
2.7 KiB
YAML
name: Security Scan
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
branches:
|
|
- 'main'
|
|
- '!oss-merge-main*'
|
|
|
|
jobs:
|
|
scan:
|
|
runs-on: ['linux', 'large']
|
|
if: ${{ github.actor != 'dependabot[bot]' || github.actor != 'hc-github-team-secure-vault-core' }}
|
|
steps:
|
|
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
|
|
|
|
- name: Set up Go
|
|
uses: actions/setup-go@fac708d6674e30b6ba41289acaab6d4b75aa0753 # v4.0.1
|
|
with:
|
|
cache: false # save cache space for vault builds: https://github.com/hashicorp/vault/pull/21764
|
|
go-version-file: .go-version
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@bd6b4b6205c4dbad673328db7b31b7fab9e241c0 # v4.6.1
|
|
with:
|
|
python-version: 3.x
|
|
|
|
- name: Clone Security Scanner repo
|
|
uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
|
|
with:
|
|
repository: hashicorp/security-scanner
|
|
token: ${{ secrets.HASHIBOT_PRODSEC_GITHUB_TOKEN }}
|
|
path: security-scanner
|
|
ref: 52d94588851f38a416f11c1e727131b3c8b0dd4d
|
|
|
|
- name: Install dependencies
|
|
shell: bash
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
run: |
|
|
mkdir "$HOME/.bin"
|
|
cd "$GITHUB_WORKSPACE/security-scanner/pkg/sdk/examples/scan-plugin-semgrep"
|
|
go build -o scan-plugin-semgrep .
|
|
mv scan-plugin-semgrep "$HOME/.bin"
|
|
|
|
cd "$GITHUB_WORKSPACE/security-scanner/pkg/sdk/examples/scan-plugin-codeql"
|
|
go build -o scan-plugin-codeql .
|
|
mv scan-plugin-codeql "$HOME/.bin"
|
|
|
|
# Semgrep
|
|
python3 -m pip install semgrep
|
|
|
|
# CodeQL
|
|
LATEST=$(gh release list --repo https://github.com/github/codeql-action | cut -f 3 | sort --version-sort | tail -n1)
|
|
gh release download --repo https://github.com/github/codeql-action --pattern codeql-bundle-linux64.tar.gz "$LATEST"
|
|
tar xf codeql-bundle-linux64.tar.gz -C "$HOME/.bin"
|
|
|
|
# Add to PATH
|
|
echo "$HOME/.bin" >> "$GITHUB_PATH"
|
|
echo "$HOME/.bin/codeql" >> "$GITHUB_PATH"
|
|
|
|
- name: Scan
|
|
id: scan
|
|
uses: ./security-scanner
|
|
# env:
|
|
# Note: this _should_ work, but causes some issues with Semgrep.
|
|
# Instead, rely on filtering in the SARIF Output step.
|
|
#SEMGREP_BASELINE_REF: ${{ github.base_ref }}
|
|
with:
|
|
repository: "$PWD"
|
|
cache-build: true
|
|
cache-go-modules: false
|
|
|
|
- name: SARIF Output
|
|
shell: bash
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
run: |
|
|
cat results.sarif
|
|
|
|
- name: Upload SARIF file
|
|
uses: github/codeql-action/upload-sarif@46a6823b81f2d7c67ddf123851eea88365bc8a67 # codeql-bundle-v2.13.5
|
|
with:
|
|
sarif_file: results.sarif
|